Stan Runge
031ee4baf2
All checks were successful
Deploy production / deploy (push) Successful in 1m10s
40 lines
7.0 KiB
Markdown
40 lines
7.0 KiB
Markdown
# stats collector
|
||
|
||
Host-side collector for the public [/stats](https://stanrunge.dev/stats) section. It pulls open
|
||
statistics into the `stats` schema with `scripts/stats-fetch.ts`:
|
||
|
||
| source | timer | schedule (Europe/Amsterdam) | what |
|
||
| ---------------- | ---------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| `cbs` | `stats-cbs.timer` | nightly 05:40 | CBS StatLine OData v4: checks every table's `Modified`, re-pulls changed ones, flags discontinued tables (base-year swaps) |
|
||
| `eurostat` | `stats-eurostat.timer` | Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP, government finance…) |
|
||
| `fred` | `stats-fred.timer` | daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices) |
|
||
| `bis` | `stats-bis.timer` | daily 07:50 | BIS central bank policy rates, monthly + daily |
|
||
| `worldbank` | `stats-worldbank.timer` | Mondays 04:15 | World Bank WDI indicators for the compared countries |
|
||
| `imf` | `stats-imf.timer` | 1st of the month 04:05 | IMF WEO (DataMapper), stored per WEO vintage with forecast flags |
|
||
| `owid` | `stats-owid.timer` | 2nd of the month 04:35 | Our World in Data: Maddison GDP per capita, energy dataset |
|
||
| `energycharts` | `stats-energycharts.timer` | hourly at :19 | Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; `--backfill` from 2015) |
|
||
| `rijksfinancien` | `stats-rijksfinancien.timer` | 3rd of the month 05:05 | Rijksfinanciën budget tables, last year to next year (`--backfill` from 2015) |
|
||
| `ustreasury` | `stats-ustreasury.timer` | daily 23:25 | US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; `--backfill` all) |
|
||
| `usaspending` | `stats-usaspending.timer` | Sundays 06:10 | USAspending budgetary resources per agency and fiscal year |
|
||
| `eponline` | `stats-eponline.timer` | 3rd of the month 05:20 | EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs `EP_ONLINE_API_KEY` in `~/.config/stats/ep-online.env` (mode 600), which the runner exports for this source only |
|
||
| `cbscw` | `stats-cbscw.timer` | Tuesdays 05:50 | CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (`stats.cw_cbs_file`); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx |
|
||
| `cwintl` | `stats-cwintl.timer` | Wednesdays 05:50 | Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat `ext_lt_maineu`) runs with the `eurostat` timer |
|
||
| – | – | manual | CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into `stats.cw_cbcs_bop` (period, item, mln_ang) |
|
||
|
||
Each timer starts `stats-collector@<source>.service`, which runs
|
||
`~/.local/bin/stats-collector-run.sh <source>`: it updates a detached checkout of `origin/main`
|
||
in `~/dev/stats-runner`, reads `DB_URL` (app role) from the running app container, and runs the
|
||
fetch. Every job is logged in `stats.fetch_run` (shown as "data freshness" on /stats).
|
||
|
||
Politeness: one shared User-Agent, a per-host gap between requests (CBS 250 ms ≈ 4 req/s,
|
||
OECD 3.1 s ≈ 19/min, FRED/BIS 1 s), retries with backoff on 429/5xx.
|
||
|
||
```sh
|
||
bash deploy/stats-collector/install.sh # install units + enable timers
|
||
systemctl --user start stats-collector@fred.service # run one source now
|
||
# long backfills: detached, survive SSH drops; logs in journalctl --user -u stats-backfill-cbs
|
||
systemd-run --user --unit=stats-backfill-cbs ~/.local/bin/stats-collector-run.sh cbs --backfill
|
||
```
|
||
|
||
Never touches the `energy` schema or ned-collector.
|