stats: jobs & cost of living theme (unemployment by age, vacancies, JOLTS, graduates, job quality, wages, price levels, housing), grad view cards, Beveridge curve; jobs collector (OECD SDMX, Big Mac index, derived series) + Eurostat/World Bank/FRED jobs series, stats-jobs.timer
All checks were successful
Deploy production / deploy (push) Successful in 1m20s
All checks were successful
Deploy production / deploy (push) Successful in 1m20s
This commit is contained in:
@@ -3,33 +3,34 @@
|
||||
Host-side collector for the public [/stats](https://stanrunge.dev/stats) section. It pulls open
|
||||
statistics into the `stats` schema with `scripts/stats-fetch.ts`:
|
||||
|
||||
| source | timer | schedule (Europe/Amsterdam) | what |
|
||||
| ---------------- | ---------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `cbs` | `stats-cbs.timer` | nightly 05:40 | CBS StatLine OData v4: checks every table's `Modified`, re-pulls changed ones, flags discontinued tables (base-year swaps) |
|
||||
| `eurostat` | `stats-eurostat.timer` | Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP index 2015 = 100 + annual rates by ECOICOP 2 category (`prc_hicp_minr`), government finance…) |
|
||||
| `fred` | `stats-fred.timer` | daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices, US CPI by category) |
|
||||
| `bis` | `stats-bis.timer` | daily 07:50 | BIS central bank policy rates (monthly + daily) and long CPI series `WS_LONG_CPI` (index 2010 = 100) for the core-7 and comparison countries |
|
||||
| `worldbank` | `stats-worldbank.timer` | Mondays 04:15 | World Bank WDI indicators for the compared countries |
|
||||
| `oecd` | `stats-oecd.timer` | Thursdays 06:35 | OECD Data Explorer SDMX CSV (no key): CPI index by category for Korea, Japan (ends 2021-06) and China (total + food), 10-year government bond yields |
|
||||
| `statcan` | `stats-statcan.timer` | daily 16:10 | Statistics Canada WDS table 18-10-0004-01: CPI all items, food, shelter, energy, core (2002 = 100) |
|
||||
| `imf` | `stats-imf.timer` | 1st of the month 04:05 | IMF WEO (DataMapper), stored per WEO vintage with forecast flags |
|
||||
| `owid` | `stats-owid.timer` | 2nd of the month 04:35 | Our World in Data: Maddison GDP per capita, energy dataset |
|
||||
| `energycharts` | `stats-energycharts.timer` | hourly at :19 | Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; `--backfill` from 2015) |
|
||||
| `rijksfinancien` | `stats-rijksfinancien.timer` | 3rd of the month 05:05 | Rijksfinanciën budget tables, last year to next year (`--backfill` from 2015) |
|
||||
| `ustreasury` | `stats-ustreasury.timer` | daily 23:25 | US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; `--backfill` all) |
|
||||
| `usaspending` | `stats-usaspending.timer` | Sundays 06:10 | USAspending budgetary resources per agency and fiscal year |
|
||||
| `eponline` | `stats-eponline.timer` | 3rd of the month 05:20 | EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs `EP_ONLINE_API_KEY` in `~/.config/stats/ep-online.env` (mode 600), which the runner exports for this source only |
|
||||
| `elections` | `stats-elections.timer` | 15th of the month 04:50 | A14: Kiesraad Tweede Kamer results 2010→ (data.overheid.nl CSV + EML zips read with HTTP ranges; national, municipality, Gouda polling stations), CBS 37278 (1918–2006), MIT US presidential by state. Finished elections are skipped unless `--force` |
|
||||
| `parties` | `stats-parties.timer` | quarterly, 16th 04:40 | A15: CHES trend file + V-Party (RData via `~/.local/share/stats-py`, pyreadr) linked with Party Facts into `wo_party`/`wo_party_position` |
|
||||
| `wikipolls` | `stats-wikipolls.timer` | daily 08:10 | A16: Wikipedia poll tables (NL seats, Trump approval, Lee Jae-myung approval, KR party polls); wikitext snapshot per revision |
|
||||
| `gtrends` | `stats-gtrends.timer` | daily 21:45 | A16: Google Trends trending-now RSS for NL, US, KR |
|
||||
| `gdelt` | `stats-gdelt.timer` | nightly 02:30 | A16: GDELT DOC timelinevol, 10 countries × 12 themes, last 30 days, ≥5.5 s between requests |
|
||||
| `eurobarometer` | `stats-eurobarometer.timer` | 20th of the month 05:35 | A16: Standard Eurobarometer volume A xlsx (QA3/QA4/QA5 most important issues, trust in institutions); new waves only |
|
||||
| `whr` | `stats-whr.timer` | 21st of the month 05:10 | A16: World Happiness Report Figure 2.1 (life evaluation and components, 2011→) |
|
||||
| `cbscw` | `stats-cbscw.timer` | Tuesdays 05:50 | CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (`stats.cw_cbs_file`); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx |
|
||||
| `cwintl` | `stats-cwintl.timer` | Wednesdays 05:50 | Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat `ext_lt_maineu`) runs with the `eurostat` timer |
|
||||
| – | – | manual | CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into `stats.cw_cbcs_bop` (period, item, mln_ang) |
|
||||
| – | – | manual | China CPI by category beyond food (NBS, data.stats.gov.cn) has no stable no-key API: not collected. Japan e-Stat needs an app id: skipped |
|
||||
| source | timer | schedule (Europe/Amsterdam) | what |
|
||||
| ---------------- | ---------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `cbs` | `stats-cbs.timer` | nightly 05:40 | CBS StatLine OData v4: checks every table's `Modified`, re-pulls changed ones, flags discontinued tables (base-year swaps) |
|
||||
| `eurostat` | `stats-eurostat.timer` | Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP index 2015 = 100 + annual rates by ECOICOP 2 category (`prc_hicp_minr`), government finance…) |
|
||||
| `fred` | `stats-fred.timer` | daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices, US CPI by category) |
|
||||
| `bis` | `stats-bis.timer` | daily 07:50 | BIS central bank policy rates (monthly + daily) and long CPI series `WS_LONG_CPI` (index 2010 = 100) for the core-7 and comparison countries |
|
||||
| `worldbank` | `stats-worldbank.timer` | Mondays 04:15 | World Bank WDI indicators for the compared countries |
|
||||
| `oecd` | `stats-oecd.timer` | Thursdays 06:35 | OECD Data Explorer SDMX CSV (no key): CPI index by category for Korea, Japan (ends 2021-06) and China (total + food), 10-year government bond yields |
|
||||
| `statcan` | `stats-statcan.timer` | daily 16:10 | Statistics Canada WDS table 18-10-0004-01: CPI all items, food, shelter, energy, core (2002 = 100) |
|
||||
| `imf` | `stats-imf.timer` | 1st of the month 04:05 | IMF WEO (DataMapper), stored per WEO vintage with forecast flags |
|
||||
| `owid` | `stats-owid.timer` | 2nd of the month 04:35 | Our World in Data: Maddison GDP per capita, energy dataset |
|
||||
| `energycharts` | `stats-energycharts.timer` | hourly at :19 | Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; `--backfill` from 2015) |
|
||||
| `rijksfinancien` | `stats-rijksfinancien.timer` | 3rd of the month 05:05 | Rijksfinanciën budget tables, last year to next year (`--backfill` from 2015) |
|
||||
| `ustreasury` | `stats-ustreasury.timer` | daily 23:25 | US Treasury MTS receipts/outlays + Debt to the Penny (last 400 days; `--backfill` all) |
|
||||
| `usaspending` | `stats-usaspending.timer` | Sundays 06:10 | USAspending budgetary resources per agency and fiscal year |
|
||||
| `eponline` | `stats-eponline.timer` | 3rd of the month 05:20 | EP-Online monthly total file (RVO energy labels): homes per label for NL, Gouda and Gouda PC4s; aggregates only. Needs `EP_ONLINE_API_KEY` in `~/.config/stats/ep-online.env` (mode 600), which the runner exports for this source only |
|
||||
| `elections` | `stats-elections.timer` | 15th of the month 04:50 | A14: Kiesraad Tweede Kamer results 2010→ (data.overheid.nl CSV + EML zips read with HTTP ranges; national, municipality, Gouda polling stations), CBS 37278 (1918–2006), MIT US presidential by state. Finished elections are skipped unless `--force` |
|
||||
| `parties` | `stats-parties.timer` | quarterly, 16th 04:40 | A15: CHES trend file + V-Party (RData via `~/.local/share/stats-py`, pyreadr) linked with Party Facts into `wo_party`/`wo_party_position` |
|
||||
| `wikipolls` | `stats-wikipolls.timer` | daily 08:10 | A16: Wikipedia poll tables (NL seats, Trump approval, Lee Jae-myung approval, KR party polls); wikitext snapshot per revision |
|
||||
| `gtrends` | `stats-gtrends.timer` | daily 21:45 | A16: Google Trends trending-now RSS for NL, US, KR |
|
||||
| `gdelt` | `stats-gdelt.timer` | nightly 02:30 | A16: GDELT DOC timelinevol, 10 countries × 12 themes, last 30 days, ≥5.5 s between requests |
|
||||
| `eurobarometer` | `stats-eurobarometer.timer` | 20th of the month 05:35 | A16: Standard Eurobarometer volume A xlsx (QA3/QA4/QA5 most important issues, trust in institutions); new waves only |
|
||||
| `whr` | `stats-whr.timer` | 21st of the month 05:10 | A16: World Happiness Report Figure 2.1 (life evaluation and components, 2011→) |
|
||||
| `cbscw` | `stats-cbscw.timer` | Tuesdays 05:50 | CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (`stats.cw_cbs_file`); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx |
|
||||
| `cwintl` | `stats-cwintl.timer` | Wednesdays 05:50 | Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat `ext_lt_maineu`) runs with the `eurostat` timer |
|
||||
| `jobs` | `stats-jobs.timer` | Thursdays 06:55 | Jobs & cost of living: OECD SDMX (average/minimum wages USD PPP, hours, 50+ h weeks, temporary/part-time shares, long-term unemployment, monthly youth unemployment, EAG graduate employment + earnings premium, house price-to-income/-rent, comparative price levels USA = 100), Economist Big Mac index (CC BY 4.0), derived US openings per unemployed (run after `fred`). Eurostat/World Bank/FRED jobs series run with their own timers |
|
||||
| – | – | manual | CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into `stats.cw_cbcs_bop` (period, item, mln_ang) |
|
||||
| – | – | manual | China CPI by category beyond food (NBS, data.stats.gov.cn) has no stable no-key API: not collected. Japan e-Stat needs an app id: skipped |
|
||||
|
||||
Each timer starts `stats-collector@<source>.service`, which runs
|
||||
`~/.local/bin/stats-collector-run.sh <source>`: it updates a detached checkout of `origin/main`
|
||||
|
||||
11
deploy/stats-collector/stats-jobs.timer
Normal file
11
deploy/stats-collector/stats-jobs.timer
Normal file
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=/stats collector: jobs & cost of living (OECD wages/hours/LFS/EAG/house prices/price levels, Big Mac index, JOLTS ratio), Thursdays
|
||||
|
||||
[Timer]
|
||||
OnCalendar=Thu *-*-* 06:55:00 Europe/Amsterdam
|
||||
RandomizedDelaySec=10min
|
||||
Persistent=true
|
||||
Unit=stats-collector@jobs.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user