stan/website
1
0
Fork 0
Code Issues 13 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
030f3c08ee42de5ae9ecad500995bae8512abe62
website/deploy/stats-collector
History
Stan Runge 031ee4baf2
All checks were successful
Deploy production / deploy (push) Successful in 1m10s
Details
stats §8: Curaçao from ILOSTAT (SDMX), OWID, UN SDG 531, UN Comtrade 531 (cwintl + weekly timer), EU trade with CW (Eurostat ext_lt_maineu); CBS Curaçao LFS file pick fix; Curaçao detail panels on /stats/countries/CUW
2026-10-25 21:47:06 +00:00
..
install.sh
stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
2026-10-25 20:36:00 +00:00
README.md
stats §8: Curaçao from ILOSTAT (SDMX), OWID, UN SDG 531, UN Comtrade 531 (cwintl + weekly timer), EU trade with CW (Eurostat ext_lt_maineu); CBS Curaçao LFS file pick fix; Curaçao detail panels on /stats/countries/CUW
2026-10-25 21:47:06 +00:00
stats-bis.timer
stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
2026-10-25 20:36:00 +00:00
stats-cbs.timer
stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
2026-10-25 20:36:00 +00:00
stats-cbscw.timer
stats §8: CBS Curaçao collector (topic-page scrape, sha256 manifest, CPI since 1970 from PDF, GDP by industry, government finance, population since 1900, LFS, tourism) into cw_cbs_* + stats.obs region CUW; cw_cbcs_bop for manual uploads; weekly timer
2026-10-25 21:38:30 +00:00
stats-collector-run.sh
stats B11: EP-Online energy labels (monthly total file → homes per label for NL, Gouda, Gouda PC4s; aggregates only), stats.ep_label_agg, monthly timer with key from ~/.config/stats/ep-online.env, charts on /stats/energy and /stats/gouda
2026-10-25 21:28:37 +00:00
stats-collector@.service
stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
2026-10-25 20:36:00 +00:00
stats-cwintl.timer
stats §8: Curaçao from ILOSTAT (SDMX), OWID, UN SDG 531, UN Comtrade 531 (cwintl + weekly timer), EU trade with CW (Eurostat ext_lt_maineu); CBS Curaçao LFS file pick fix; Curaçao detail panels on /stats/countries/CUW
2026-10-25 21:47:06 +00:00
stats-energycharts.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00
stats-eponline.timer
stats B11: EP-Online energy labels (monthly total file → homes per label for NL, Gouda, Gouda PC4s; aggregates only), stats.ep_label_agg, monthly timer with key from ~/.config/stats/ep-online.env, charts on /stats/energy and /stats/gouda
2026-10-25 21:28:37 +00:00
stats-eurostat.timer
stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
2026-10-25 20:36:00 +00:00
stats-fred.timer
stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
2026-10-25 20:36:00 +00:00
stats-imf.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00
stats-owid.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00
stats-rijksfinancien.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00
stats-usaspending.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00
stats-ustreasury.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00
stats-worldbank.timer
stats step 2: World Bank, IMF WEO, OWID, Energy-Charts, Rijksfinanciën, US Treasury, USAspending collectors + tables, timers
2026-10-25 21:05:51 +00:00

README.md

stats collector

Host-side collector for the public /stats section. It pulls open statistics into the stats schema with scripts/stats-fetch.ts:

source timer schedule (Europe/Amsterdam) what
cbs stats-cbs.timer nightly 05:40 CBS StatLine OData v4: checks every table's Modified, re-pulls changed ones, flags discontinued tables (base-year swaps)
eurostat stats-eurostat.timer Mondays 06:20 Eurostat JSON-stat datasets (energy prices, yields, HICP, government finance…)
fred stats-fred.timer daily 07:20 FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices)
bis stats-bis.timer daily 07:50 BIS central bank policy rates, monthly + daily
worldbank stats-worldbank.timer Mondays 04:15 World Bank WDI indicators for the compared countries
imf stats-imf.timer 1st of the month 04:05 IMF WEO (DataMapper), stored per WEO vintage with forecast flags
owid stats-owid.timer 2nd of the month 04:35 Our World in Data: Maddison GDP per capita, energy dataset
energycharts stats-energycharts.timer hourly at :19 Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; --backfill from 2015)
rijksfinancien stats-rijksfinancien.timer 3rd of the month 05:05 Rijksfinanciën budget tables, last year to next year (--backfill from 2015)
ustreasury stats-ustreasury.timer daily 23:25 US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; --backfill all)
usaspending stats-usaspending.timer Sundays 06:10 USAspending budgetary resources per agency and fiscal year
eponline stats-eponline.timer 3rd of the month 05:20 EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs EP_ONLINE_API_KEY in ~/.config/stats/ep-online.env (mode 600), which the runner exports for this source only
cbscw stats-cbscw.timer Tuesdays 05:50 CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (stats.cw_cbs_file); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx
cwintl stats-cwintl.timer Wednesdays 05:50 Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat ext_lt_maineu) runs with the eurostat timer
– – manual CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into stats.cw_cbcs_bop (period, item, mln_ang)

Each timer starts stats-collector@<source>.service, which runs ~/.local/bin/stats-collector-run.sh <source>: it updates a detached checkout of origin/main in ~/dev/stats-runner, reads DB_URL (app role) from the running app container, and runs the fetch. Every job is logged in stats.fetch_run (shown as "data freshness" on /stats).

Politeness: one shared User-Agent, a per-host gap between requests (CBS 250 ms ≈ 4 req/s, OECD 3.1 s ≈ 19/min, FRED/BIS 1 s), retries with backoff on 429/5xx.

bash deploy/stats-collector/install.sh                     # install units + enable timers
systemctl --user start stats-collector@fred.service        # run one source now
# long backfills: detached, survive SSH drops; logs in journalctl --user -u stats-backfill-cbs
systemd-run --user --unit=stats-backfill-cbs ~/.local/bin/stats-collector-run.sh cbs --backfill

Never touches the energy schema or ned-collector.