website/deploy/stats-collector

stats collector

Host-side collector for the public /stats section. It pulls open statistics (no API keys) into the stats schema with scripts/stats-fetch.ts:

source	timer	schedule (Europe/Amsterdam)	what
cbs	stats-cbs.timer	nightly 05:40	CBS StatLine OData v4: checks every table's Modified, re-pulls changed ones, flags discontinued tables (base-year swaps)
eurostat	stats-eurostat.timer	Mondays 06:20	Eurostat JSON-stat datasets (energy prices, yields, HICP, government finance…)
fred	stats-fred.timer	daily 07:20	FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices)
bis	stats-bis.timer	daily 07:50	BIS central bank policy rates, monthly + daily

Each timer starts stats-collector@<source>.service, which runs ~/.local/bin/stats-collector-run.sh <source>: it updates a detached checkout of origin/main in ~/dev/stats-runner, reads DB_URL (app role) from the running app container, and runs the fetch. Every job is logged in stats.fetch_run (shown as "data freshness" on /stats).

Politeness: one shared User-Agent, a per-host gap between requests (CBS 250 ms ≈ 4 req/s, OECD 3.1 s ≈ 19/min, FRED/BIS 1 s), retries with backoff on 429/5xx.

bash deploy/stats-collector/install.sh                     # install units + enable timers
systemctl --user start stats-collector@fred.service        # run one source now
# long backfills: detached, survive SSH drops; logs in journalctl --user -u stats-backfill-cbs
systemd-run --user --unit=stats-backfill-cbs ~/.local/bin/stats-collector-run.sh cbs --backfill

Never touches the energy schema or ned-collector.