stats step 3: elections, party positions and opinion feeds (schema, collectors, timers); world-opinions data moved into stats.wo_*
All checks were successful
Deploy production / deploy (push) Successful in 1m12s

This commit is contained in:
2026-10-25 22:16:25 +00:00
parent 030f3c08ee
commit 42c9351ebd
37 changed files with 75721 additions and 892 deletions

View File

@@ -17,6 +17,13 @@ statistics into the `stats` schema with `scripts/stats-fetch.ts`:
| `ustreasury` | `stats-ustreasury.timer` | daily 23:25 | US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; `--backfill` all) |
| `usaspending` | `stats-usaspending.timer` | Sundays 06:10 | USAspending budgetary resources per agency and fiscal year |
| `eponline` | `stats-eponline.timer` | 3rd of the month 05:20 | EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs `EP_ONLINE_API_KEY` in `~/.config/stats/ep-online.env` (mode 600), which the runner exports for this source only |
| `elections` | `stats-elections.timer` | 15th of the month 04:50 | A14: Kiesraad Tweede Kamer results 2010→ (data.overheid.nl CSV + EML zips read with HTTP ranges; national, municipality, Gouda polling stations), CBS 37278 (1918–2006), MIT US presidential by state. Finished elections are skipped unless `--force` |
| `parties` | `stats-parties.timer` | quarterly, 16th 04:40 | A15: CHES trend file + V-Party (RData via `~/.local/share/stats-py`, pyreadr) linked with Party Facts into `wo_party`/`wo_party_position` |
| `wikipolls` | `stats-wikipolls.timer` | daily 08:10 | A16: Wikipedia poll tables (NL seats, Trump approval, Lee Jae-myung approval, KR party polls); wikitext snapshot per revision |
| `gtrends` | `stats-gtrends.timer` | daily 21:45 | A16: Google Trends trending-now RSS for NL, US, KR |
| `gdelt` | `stats-gdelt.timer` | nightly 02:30 | A16: GDELT DOC timelinevol, 10 countries × 12 themes, last 30 days, ≥5.5 s between requests |
| `eurobarometer` | `stats-eurobarometer.timer` | 20th of the month 05:35 | A16: Standard Eurobarometer volume A xlsx (QA3/QA4/QA5 most important issues, trust in institutions); new waves only |
| `whr` | `stats-whr.timer` | 21st of the month 05:10 | A16: World Happiness Report Figure 2.1 (life evaluation and components, 2011→) |
| `cbscw` | `stats-cbscw.timer` | Tuesdays 05:50 | CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (`stats.cw_cbs_file`); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx |
| `cwintl` | `stats-cwintl.timer` | Wednesdays 05:50 | Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat `ext_lt_maineu`) runs with the `eurostat` timer |
| – | – | manual | CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into `stats.cw_cbcs_bop` (period, item, mln_ang) |