stats collector
Host-side collector for the public /stats section. It pulls open
statistics into the stats schema with scripts/stats-fetch.ts:
| source | timer | schedule (Europe/Amsterdam) | what |
|---|---|---|---|
cbs |
stats-cbs.timer |
nightly 05:40 | CBS StatLine OData v4: checks every table's Modified, re-pulls changed ones, flags discontinued tables (base-year swaps) |
eurostat |
stats-eurostat.timer |
Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP, government finance…) |
fred |
stats-fred.timer |
daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices) |
bis |
stats-bis.timer |
daily 07:50 | BIS central bank policy rates, monthly + daily |
worldbank |
stats-worldbank.timer |
Mondays 04:15 | World Bank WDI indicators for the compared countries |
imf |
stats-imf.timer |
1st of the month 04:05 | IMF WEO (DataMapper), stored per WEO vintage with forecast flags |
owid |
stats-owid.timer |
2nd of the month 04:35 | Our World in Data: Maddison GDP per capita, energy dataset |
energycharts |
stats-energycharts.timer |
hourly at :19 | Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; --backfill from 2015) |
rijksfinancien |
stats-rijksfinancien.timer |
3rd of the month 05:05 | Rijksfinanciën budget tables, last year to next year (--backfill from 2015) |
ustreasury |
stats-ustreasury.timer |
daily 23:25 | US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; --backfill all) |
usaspending |
stats-usaspending.timer |
Sundays 06:10 | USAspending budgetary resources per agency and fiscal year |
eponline |
stats-eponline.timer |
3rd of the month 05:20 | EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs EP_ONLINE_API_KEY in ~/.config/stats/ep-online.env (mode 600), which the runner exports for this source only |
Each timer starts stats-collector@<source>.service, which runs
~/.local/bin/stats-collector-run.sh <source>: it updates a detached checkout of origin/main
in ~/dev/stats-runner, reads DB_URL (app role) from the running app container, and runs the
fetch. Every job is logged in stats.fetch_run (shown as "data freshness" on /stats).
Politeness: one shared User-Agent, a per-host gap between requests (CBS 250 ms ≈ 4 req/s, OECD 3.1 s ≈ 19/min, FRED/BIS 1 s), retries with backoff on 429/5xx.
bash deploy/stats-collector/install.sh # install units + enable timers
systemctl --user start stats-collector@fred.service # run one source now
# long backfills: detached, survive SSH drops; logs in journalctl --user -u stats-backfill-cbs
systemd-run --user --unit=stats-backfill-cbs ~/.local/bin/stats-collector-run.sh cbs --backfill
Never touches the energy schema or ned-collector.