website/deploy/stats-collector/README.md

stats collector

Host-side collector for the public /stats section. It pulls open statistics into the stats schema with scripts/stats-fetch.ts:

source	timer	schedule (Europe/Amsterdam)	what
cbs	stats-cbs.timer	nightly 05:40	CBS StatLine OData v4: checks every table's Modified, re-pulls changed ones, flags discontinued tables (base-year swaps)
eurostat	stats-eurostat.timer	Mondays 06:20	Eurostat JSON-stat datasets (energy prices, yields, HICP index 2015 = 100 + annual rates by ECOICOP 2 category (prc_hicp_minr), government finance…)
fred	stats-fred.timer	daily 07:20	FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices, US CPI by category)
bis	stats-bis.timer	daily 07:50	BIS central bank policy rates (monthly + daily) and long CPI series WS_LONG_CPI (index 2010 = 100) for the core-7 and comparison countries
worldbank	stats-worldbank.timer	Mondays 04:15	World Bank WDI indicators for the compared countries
oecd	stats-oecd.timer	Thursdays 06:35	OECD Data Explorer SDMX CSV (no key): CPI index by category for Korea, Japan (ends 2021-06) and China (total + food), 10-year government bond yields
statcan	stats-statcan.timer	daily 16:10	Statistics Canada WDS table 18-10-0004-01: CPI all items, food, shelter, energy, core (2002 = 100)
imf	stats-imf.timer	1st of the month 04:05	IMF WEO (DataMapper), stored per WEO vintage with forecast flags
owid	stats-owid.timer	2nd of the month 04:35	Our World in Data: Maddison GDP per capita, energy dataset
energycharts	stats-energycharts.timer	hourly at :19	Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; --backfill from 2015)
rijksfinancien	stats-rijksfinancien.timer	3rd of the month 05:05	Rijksfinanciën budget tables, last year to next year (--backfill from 2015)
ustreasury	stats-ustreasury.timer	daily 23:25	US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; --backfill all)
usaspending	stats-usaspending.timer	Sundays 06:10	USAspending budgetary resources per agency and fiscal year
eponline	stats-eponline.timer	3rd of the month 05:20	EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs EP_ONLINE_API_KEY in ~/.config/stats/ep-online.env (mode 600), which the runner exports for this source only
elections	stats-elections.timer	15th of the month 04:50	A14: Kiesraad Tweede Kamer results 2010→ (data.overheid.nl CSV + EML zips read with HTTP ranges; national, municipality, Gouda polling stations), CBS 37278 (1918–2006), MIT US presidential by state. Finished elections are skipped unless --force
parties	stats-parties.timer	quarterly, 16th 04:40	A15: CHES trend file + V-Party (RData via ~/.local/share/stats-py, pyreadr) linked with Party Facts into wo_party/wo_party_position
wikipolls	stats-wikipolls.timer	daily 08:10	A16: Wikipedia poll tables (NL seats, Trump approval, Lee Jae-myung approval, KR party polls); wikitext snapshot per revision
gtrends	stats-gtrends.timer	daily 21:45	A16: Google Trends trending-now RSS for NL, US, KR
gdelt	stats-gdelt.timer	nightly 02:30	A16: GDELT DOC timelinevol, 10 countries × 12 themes, last 30 days, ≥5.5 s between requests
eurobarometer	stats-eurobarometer.timer	20th of the month 05:35	A16: Standard Eurobarometer volume A xlsx (QA3/QA4/QA5 most important issues, trust in institutions); new waves only
whr	stats-whr.timer	21st of the month 05:10	A16: World Happiness Report Figure 2.1 (life evaluation and components, 2011→)
cbscw	stats-cbscw.timer	Tuesdays 05:50	CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (stats.cw_cbs_file); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx
cwintl	stats-cwintl.timer	Wednesdays 05:50	Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat ext_lt_maineu) runs with the eurostat timer
jobs	stats-jobs.timer	Thursdays 06:55	Jobs & cost of living: OECD SDMX (average/minimum wages USD PPP, hours, 50+ h weeks, temporary/part-time shares, long-term unemployment, monthly youth unemployment, EAG graduate employment + earnings premium, house price-to-income/-rent, comparative price levels USA = 100), Economist Big Mac index (CC BY 4.0), derived US openings per unemployed (run after fred). Eurostat/World Bank/FRED jobs series run with their own timers
–	–	manual	CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into stats.cw_cbcs_bop (period, item, mln_ang)
–	–	manual	China CPI by category beyond food (NBS, data.stats.gov.cn) has no stable no-key API: not collected. Japan e-Stat needs an app id: skipped

Each timer starts stats-collector@<source>.service, which runs ~/.local/bin/stats-collector-run.sh <source>: it updates a detached checkout of origin/main in ~/dev/stats-runner, reads DB_URL (app role) from the running app container, and runs the fetch. Every job is logged in stats.fetch_run (shown as "data freshness" on /stats).

Politeness: one shared User-Agent, a per-host gap between requests (CBS 250 ms ≈ 4 req/s, OECD 3.1 s ≈ 19/min, FRED/BIS 1 s), retries with backoff on 429/5xx.

bash deploy/stats-collector/install.sh                     # install units + enable timers
systemctl --user start stats-collector@fred.service        # run one source now
# long backfills: detached, survive SSH drops; logs in journalctl --user -u stats-backfill-cbs
systemd-run --user --unit=stats-backfill-cbs ~/.local/bin/stats-collector-run.sh cbs --backfill

Never touches the energy schema or ned-collector.