From 45841d6f6cf024b1377243533933b59f225c95d3 Mon Sep 17 00:00:00 2001 From: Stan Runge Date: Mon, 26 Oct 2026 00:34:27 +0000 Subject: [PATCH] stats: core-7 economy page with CPI index levels, rebase and categories - /stats/economy defaults multi-country charts to CAN KOR CHN JPN USA NLD CUW, with add-a-country chips and an EU 27 comparison line - CPI index levels rebased to a chosen base year, index/YoY toggle, categories (food, energy, housing, core) where the source has them - new no-key collectors: oecd (CPI categories, 10y yields), statcan (CPI); bis WS_LONG_CPI, FRED US CPI categories, Eurostat prc_hicp_minr, WB FP.CPI.TOTL - Canada added as a region; CBS Curacao fallbacks for growth/unemployment - timers stats-oecd and stats-statcan --- deploy/stats-collector/README.md | 9 +- deploy/stats-collector/stats-oecd.timer | 11 + deploy/stats-collector/stats-statcan.timer | 11 + scripts/stats-fetch.ts | 12 +- scripts/stats/bis.ts | 53 ++- scripts/stats/common.ts | 26 +- scripts/stats/eurostat.ts | 10 +- scripts/stats/fred.ts | 15 + scripts/stats/oecd.ts | 129 +++++++ scripts/stats/statcan.ts | 57 +++ scripts/stats/worldbank.ts | 1 + src/lib/stats/CountryPicker.svelte | 96 +++++ src/lib/stats/countries.ts | 3 + src/lib/stats/econ.test.ts | 27 ++ src/lib/stats/econ.ts | 151 ++++++++ src/lib/stats/regions.ts | 2 + src/routes/stats/economy/+page.server.ts | 29 +- src/routes/stats/economy/+page.svelte | 406 +++++++++++++++------ 18 files changed, 926 insertions(+), 122 deletions(-) create mode 100644 deploy/stats-collector/stats-oecd.timer create mode 100644 deploy/stats-collector/stats-statcan.timer create mode 100644 scripts/stats/oecd.ts create mode 100644 scripts/stats/statcan.ts create mode 100644 src/lib/stats/CountryPicker.svelte create mode 100644 src/lib/stats/econ.test.ts create mode 100644 src/lib/stats/econ.ts diff --git a/deploy/stats-collector/README.md b/deploy/stats-collector/README.md index a3d850e..bfb37d1 100644 --- a/deploy/stats-collector/README.md +++ b/deploy/stats-collector/README.md @@ -6,10 +6,12 @@ statistics into the `stats` schema with `scripts/stats-fetch.ts`: | source | timer | schedule (Europe/Amsterdam) | what | | ---------------- | ---------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cbs` | `stats-cbs.timer` | nightly 05:40 | CBS StatLine OData v4: checks every table's `Modified`, re-pulls changed ones, flags discontinued tables (base-year swaps) | -| `eurostat` | `stats-eurostat.timer` | Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP, government finance…) | -| `fred` | `stats-fred.timer` | daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices) | -| `bis` | `stats-bis.timer` | daily 07:50 | BIS central bank policy rates, monthly + daily | +| `eurostat` | `stats-eurostat.timer` | Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP index 2015 = 100 + annual rates by ECOICOP 2 category (`prc_hicp_minr`), government finance…) | +| `fred` | `stats-fred.timer` | daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices, US CPI by category) | +| `bis` | `stats-bis.timer` | daily 07:50 | BIS central bank policy rates (monthly + daily) and long CPI series `WS_LONG_CPI` (index 2010 = 100) for the core-7 and comparison countries | | `worldbank` | `stats-worldbank.timer` | Mondays 04:15 | World Bank WDI indicators for the compared countries | +| `oecd` | `stats-oecd.timer` | Thursdays 06:35 | OECD Data Explorer SDMX CSV (no key): CPI index by category for Korea, Japan (ends 2021-06) and China (total + food), 10-year government bond yields | +| `statcan` | `stats-statcan.timer` | daily 16:10 | Statistics Canada WDS table 18-10-0004-01: CPI all items, food, shelter, energy, core (2002 = 100) | | `imf` | `stats-imf.timer` | 1st of the month 04:05 | IMF WEO (DataMapper), stored per WEO vintage with forecast flags | | `owid` | `stats-owid.timer` | 2nd of the month 04:35 | Our World in Data: Maddison GDP per capita, energy dataset | | `energycharts` | `stats-energycharts.timer` | hourly at :19 | Energy-Charts NL day-ahead prices + cross-border flows (last 3 days; `--backfill` from 2015) | @@ -27,6 +29,7 @@ statistics into the `stats` schema with `scripts/stats-fetch.ts`: | `cbscw` | `stats-cbscw.timer` | Tuesdays 05:50 | CBS Curaçao: scrapes the topic pages (cbs.cw), matches each table's file by regex, re-imports only when its sha256 changes (`stats.cw_cbs_file`); CPI from PDF (pdfjs), GDP, government finance, population since 1900, LFS, tourism from xlsx | | `cwintl` | `stats-cwintl.timer` | Wednesdays 05:50 | Curaçao from international APIs: ILOSTAT SDMX (labour, with retries), OWID grapher CSVs (population, life expectancy, median age, CO₂), UN SDG area 531, UN Comtrade public preview reporter 531. EU trade with Curaçao (Eurostat `ext_lt_maineu`) runs with the `eurostat` timer | | – | – | manual | CBCS (centralbank.cw) blocks scripts with a Cloudflare challenge: download the BoP xlsx by hand and load it into `stats.cw_cbcs_bop` (period, item, mln_ang) | +| – | – | manual | China CPI by category beyond food (NBS, data.stats.gov.cn) has no stable no-key API: not collected. Japan e-Stat needs an app id: skipped | Each timer starts `stats-collector@.service`, which runs `~/.local/bin/stats-collector-run.sh `: it updates a detached checkout of `origin/main` diff --git a/deploy/stats-collector/stats-oecd.timer b/deploy/stats-collector/stats-oecd.timer new file mode 100644 index 0000000..b348835 --- /dev/null +++ b/deploy/stats-collector/stats-oecd.timer @@ -0,0 +1,11 @@ +[Unit] +Description=/stats collector: OECD SDMX CPI categories and 10-year yields, Thursdays + +[Timer] +OnCalendar=Thu *-*-* 06:35:00 Europe/Amsterdam +RandomizedDelaySec=10min +Persistent=true +Unit=stats-collector@oecd.service + +[Install] +WantedBy=timers.target diff --git a/deploy/stats-collector/stats-statcan.timer b/deploy/stats-collector/stats-statcan.timer new file mode 100644 index 0000000..651b722 --- /dev/null +++ b/deploy/stats-collector/stats-statcan.timer @@ -0,0 +1,11 @@ +[Unit] +Description=/stats collector: Statistics Canada CPI daily + +[Timer] +OnCalendar=*-*-* 16:10:00 Europe/Amsterdam +RandomizedDelaySec=10min +Persistent=true +Unit=stats-collector@statcan.service + +[Install] +WantedBy=timers.target diff --git a/scripts/stats-fetch.ts b/scripts/stats-fetch.ts index 2dbbc61..eea41a8 100644 --- a/scripts/stats-fetch.ts +++ b/scripts/stats-fetch.ts @@ -6,6 +6,8 @@ // bun scripts/stats-fetch.ts eurostat [--only=nrg_pc_204] // bun scripts/stats-fetch.ts fred [--only=DCOILBRENTEU] // bun scripts/stats-fetch.ts bis +// bun scripts/stats-fetch.ts oecd [--only=cpi,irlt] OECD SDMX (CPI categories, 10y yields) +// bun scripts/stats-fetch.ts statcan Statistics Canada CPI (WDS) // bun scripts/stats-fetch.ts worldbank|imf|owid [--only=…] // bun scripts/stats-fetch.ts energycharts|rijksfinancien|ustreasury [--backfill] // bun scripts/stats-fetch.ts usaspending @@ -21,6 +23,8 @@ import { cbs, normaliseCbs } from './stats/cbs'; import { eurostat } from './stats/eurostat'; import { fred } from './stats/fred'; import { bis } from './stats/bis'; +import { oecd } from './stats/oecd'; +import { statcan } from './stats/statcan'; import { worldbank } from './stats/worldbank'; import { imf } from './stats/imf'; import { owid } from './stats/owid'; @@ -63,6 +67,12 @@ try { case 'bis': await bis({ job }); break; + case 'oecd': + await oecd({ only: value('only'), job }); + break; + case 'statcan': + await statcan({ job }); + break; case 'worldbank': await worldbank({ only: value('only'), job }); break; @@ -119,7 +129,7 @@ try { break; default: throw new Error( - `Unknown source "${source}" (cbs, eurostat, fred, bis, worldbank, imf, owid, energycharts, rijksfinancien, ustreasury, usaspending, eponline, cbscw, cwintl, elections, parties, wikipolls, eurobarometer, whr, gtrends, gdelt, history, normalise-cbs)` + `Unknown source "${source}" (cbs, eurostat, fred, bis, oecd, statcan, worldbank, imf, owid, energycharts, rijksfinancien, ustreasury, usaspending, eponline, cbscw, cwintl, elections, parties, wikipolls, eurobarometer, whr, gtrends, gdelt, history, normalise-cbs)` ); } } finally { diff --git a/scripts/stats/bis.ts b/scripts/stats/bis.ts index f48fab3..9d8b52e 100644 --- a/scripts/stats/bis.ts +++ b/scripts/stats/bis.ts @@ -1,4 +1,5 @@ -// A5 BIS central bank policy rates (WS_CBPOL), monthly and daily, in one call each. +// A5 BIS central bank policy rates (WS_CBPOL), monthly and daily, in one call each; plus the BIS +// long consumer price series (WS_LONG_CPI, index 2010 = 100, monthly) → `bis:cpi:`. import { getText, log, run, sql, writeSeries, type ObsRow, type SeriesRow } from './common'; const AREAS: Record = { @@ -7,7 +8,26 @@ const AREAS: Record = { KR: { name: 'Bank of Korea', region: 'KOR' }, CN: { name: "People's Bank of China", region: 'CHN' }, GB: { name: 'Bank of England', region: 'GBR' }, - JP: { name: 'Bank of Japan', region: 'JPN' } + JP: { name: 'Bank of Japan', region: 'JPN' }, + CA: { name: 'Bank of Canada', region: 'CAN' } +}; + +/** WS_LONG_CPI areas (no Curaçao: its CPI comes from CBS Curaçao). */ +const CPI_AREAS: Record = { + CA: 'CAN', + CN: 'CHN', + JP: 'JPN', + KR: 'KOR', + US: 'USA', + NL: 'NLD', + XM: 'EA', + DE: 'DEU', + FR: 'FRA', + BE: 'BEL', + GB: 'GBR', + IN: 'IND', + BR: 'BRA', + RU: 'RUS' }; function parseCsv(text: string): Record[] { @@ -73,6 +93,35 @@ export async function bis(opts: { job: string }) { log(`bis ${freq}: ${rows.length} rows`); total += rows.length; } + // Long consumer prices: unit 628 = index, 2010 = 100. + const cpiCsv = await getText( + `https://stats.bis.org/api/v2/data/dataflow/BIS/WS_LONG_CPI/1.0/M.${Object.keys(CPI_AREAS).join('+')}.628?format=csv` + ); + const cpiObs: ObsRow[] = parseCsv(cpiCsv) + .filter((r) => r.REF_AREA in CPI_AREAS && /^\d{4}-\d{2}/.test(r.TIME_PERIOD ?? '')) + .filter((r) => r.OBS_VALUE !== '' && r.OBS_VALUE !== 'NaN') + .map((r) => ({ + series_id: `bis:cpi:${r.REF_AREA}`, + period: `${r.TIME_PERIOD.slice(0, 7)}-01`, + value: Number(r.OBS_VALUE), + status: r.OBS_STATUS && r.OBS_STATUS !== 'A' ? r.OBS_STATUS : null + })); + const cpiSeries: SeriesRow[] = Object.entries(CPI_AREAS) + .filter(([code]) => cpiObs.some((o) => o.series_id === `bis:cpi:${code}`)) + .map(([code, region]) => ({ + series_id: `bis:cpi:${code}`, + source_id: 'bis', + source_code: `WS_LONG_CPI/M.${code}.628`, + title: 'Consumer prices (BIS long series)', + unit: 'Index 2010=100', + freq: 'M', + region_code: region, + theme: 'economy', + meta: { url: 'https://data.bis.org/topics/CPI' } + })); + await writeSeries(cpiSeries, cpiObs); + log(`bis cpi: ${cpiObs.length} obs`); + total += cpiObs.length; return total; }); } diff --git a/scripts/stats/common.ts b/scripts/stats/common.ts index 9e4b154..769e2fd 100644 --- a/scripts/stats/common.ts +++ b/scripts/stats/common.ts @@ -24,6 +24,7 @@ const hostGapMs: Record = { 'fred.stlouisfed.org': 1000, 'ec.europa.eu': 500, 'stats.bis.org': 1000, + 'www150.statcan.gc.ca': 1000, 'api.worldbank.org': 1000, 'www.imf.org': 1500, 'ourworldindata.org': 2000, @@ -150,11 +151,28 @@ export const SOURCES = [ }, { source_id: 'bis', - name: 'BIS central bank policy rates', - base_url: 'https://stats.bis.org/api/v2/data/dataflow/BIS/WS_CBPOL/1.0', + name: 'BIS (central bank policy rates, long consumer price series)', + base_url: 'https://stats.bis.org/api/v2/data/dataflow/BIS', licence: 'Free reuse with attribution', key_required: false, - attribution: 'Source: BIS, central bank policy rates (WS_CBPOL)' + attribution: + 'Source: BIS, central bank policy rates (WS_CBPOL) and consumer prices (WS_LONG_CPI)' + }, + { + source_id: 'oecd', + name: 'OECD Data Explorer (SDMX)', + base_url: 'https://sdmx.oecd.org/public/rest/data', + licence: 'CC BY 4.0', + key_required: false, + attribution: 'Source: OECD Data Explorer (prices; financial market statistics), CC BY 4.0' + }, + { + source_id: 'statcan', + name: 'Statistics Canada (Web Data Service)', + base_url: 'https://www150.statcan.gc.ca/t1/wds/rest', + licence: 'Statistics Canada Open Licence', + key_required: false, + attribution: 'Source: Statistics Canada, table 18-10-0004-01 (Statistics Canada Open Licence)' }, { source_id: 'worldbank', @@ -395,6 +413,7 @@ export const REGIONS: { region_code: string; name: string; kind: string; parent: { region_code: 'AUT', name: 'Austria', kind: 'country', parent: 'EU27' }, { region_code: 'GBR', name: 'United Kingdom', kind: 'country', parent: 'WLD' }, { region_code: 'JPN', name: 'Japan', kind: 'country', parent: 'WLD' }, + { region_code: 'CAN', name: 'Canada', kind: 'country', parent: 'WLD' }, { region_code: 'IND', name: 'India', kind: 'country', parent: 'WLD' }, { region_code: 'BRA', name: 'Brazil', kind: 'country', parent: 'WLD' }, { region_code: 'RUS', name: 'Russia', kind: 'country', parent: 'WLD' }, @@ -415,6 +434,7 @@ export const COMPARE_ISO3 = [ 'KOR', 'CHN', 'CUW', + 'CAN', 'DEU', 'FRA', 'BEL', diff --git a/scripts/stats/eurostat.ts b/scripts/stats/eurostat.ts index a8761e6..753d63a 100644 --- a/scripts/stats/eurostat.ts +++ b/scripts/stats/eurostat.ts @@ -87,7 +87,15 @@ export const EUROSTAT_DATASETS: Dataset[] = [ params: {}, geos: ['NL', 'DE', 'FR', 'IT', 'BE', 'ES', 'AT', 'EA'] }, - { code: 'prc_hicp_minr', theme: 'economy', params: { coicop18: 'TOTAL', unit: 'RCH_A' } }, + { + // Rates for every geo; index levels (2015 = 100) and the main categories feed the CPI charts. + code: 'prc_hicp_minr', + theme: 'economy', + params: { + coicop18: ['TOTAL', 'CP01', 'CP04', 'NRG', 'TOT_X_NRG_FOOD'], + unit: ['RCH_A', 'I15'] + } + }, { code: 'prc_hicp_manr', theme: 'economy', params: { coicop: 'CP00', unit: 'RCH_A' } }, { code: 'prc_hpi_q', theme: 'economy', params: { purchase: 'TOTAL', unit: ['I15_Q', 'RCH_A'] } }, { code: 'road_eqr_carpda', theme: 'energy', params: { unit: 'NR' } }, diff --git a/scripts/stats/fred.ts b/scripts/stats/fred.ts index f13c973..73ed32f 100644 --- a/scripts/stats/fred.ts +++ b/scripts/stats/fred.ts @@ -155,6 +155,21 @@ export const FRED_SERIES: Fred[] = [ region: 'USA', theme: 'economy' }, + ...( + [ + ['CPIUFDSL', 'food'], + ['CPIHOSSL', 'housing'], + ['CPIENGSL', 'energy'], + ['CPILFESL', 'all items less food and energy'] + ] as const + ).map(([id, what]) => ({ + id, + title: `US consumer price index (CPI-U), ${what}`, + unit: 'Index 1982-84=100', + freq: 'M' as const, + region: 'USA', + theme: 'economy' + })), { id: 'UNRATE', title: 'US unemployment rate', diff --git a/scripts/stats/oecd.ts b/scripts/stats/oecd.ts new file mode 100644 index 0000000..1fdb820 --- /dev/null +++ b/scripts/stats/oecd.ts @@ -0,0 +1,129 @@ +// OECD Data Explorer SDMX (no key, ≤20 requests a minute): consumer price indices by category +// (DF_PRICES_ALL, 2015 = 100) for the countries without a national feed here, and long-term +// (10-year) government bond yields (DF_FINMARK). Normalised only: `oecd:cpi::` +// and `oecd:irlt:`. +import { http, log, run, writeSeries, type ObsRow, type SeriesRow } from './common'; + +const BASE = 'https://sdmx.oecd.org/public/rest/data'; +const CSV = { accept: 'application/vnd.sdmx.data+csv; charset=utf-8' }; + +/** CPI categories (OECD COICOP codes) → our short names. */ +export const OECD_CPI_CATEGORIES: Record = { + _T: { name: 'total', title: 'Consumer price index, all items' }, + CP01: { name: 'food', title: 'Consumer price index, food and non-alcoholic beverages' }, + CP04: { + name: 'housing', + title: 'Consumer price index, housing, water, electricity, gas and other fuels' + }, + CP045_0722: { name: 'energy', title: 'Consumer price index, energy' }, + _TXCP01_NRG: { name: 'core', title: 'Consumer price index, all items excluding food and energy' } +}; +/** National CPI comes from CBS, BLS (FRED), StatCan and CBS Curaçao; OECD fills KOR, JPN, CHN. */ +const CPI_AREAS = ['KOR', 'JPN', 'CHN']; +const IRLT_AREAS = [ + 'CAN', + 'JPN', + 'KOR', + 'CHN', + 'USA', + 'NLD', + 'DEU', + 'FRA', + 'GBR', + 'BEL', + 'IND', + 'BRA', + 'RUS', + 'EA20' +]; +const region = (a: string) => (a === 'EA20' ? 'EA' : a); + +function parseCsv(text: string): Record[] { + const lines = text.trim().split(/\r?\n/); + const head = lines[0].split(','); + return lines.slice(1).map((l) => Object.fromEntries(l.split(',').map((v, i) => [head[i], v]))); +} + +async function pull( + path: string, + key: (r: Record) => { id: string; region: string; code: string } | null, + meta: (code: string) => { title: string; unit: string } +) { + const rows = parseCsv(await (await http(`${BASE}/${path}`, { headers: CSV })).text()); + const series = new Map(); + const obs: ObsRow[] = []; + for (const r of rows) { + const k = key(r); + if (!k || !/^\d{4}-\d{2}$/.test(r.TIME_PERIOD ?? '') || r.OBS_VALUE === '') continue; + const value = Number(r.OBS_VALUE); + if (!Number.isFinite(value)) continue; + if (!series.has(k.id)) { + const m = meta(k.code); + series.set(k.id, { + series_id: k.id, + source_id: 'oecd', + source_code: `${path.split('/')[0]}/${k.code}`, + title: m.title, + unit: m.unit, + freq: 'M', + region_code: k.region, + theme: 'economy', + meta: { url: 'https://data-explorer.oecd.org/' } + }); + } + obs.push({ + series_id: k.id, + period: `${r.TIME_PERIOD}-01`, + value, + status: r.OBS_STATUS || null + }); + } + if (!obs.length) throw new Error(`no rows for ${path.split('?')[0]}`); + await writeSeries([...series.values()], obs); + return obs.length; +} + +export async function oecd(opts: { only?: string; job: string }) { + await run('oecd', opts.job, async (note) => { + let total = 0; + const want = (j: string) => !opts.only || opts.only.split(',').includes(j); + if (want('cpi')) + try { + const n = await pull( + `OECD.SDD.TPS,DSD_PRICES@DF_PRICES_ALL,/${CPI_AREAS.join('+')}.M.N.CPI.IX.${Object.keys(OECD_CPI_CATEGORIES).join('+')}.N._Z?startPeriod=1950-01`, + (r) => { + const c = OECD_CPI_CATEGORIES[r.EXPENDITURE]; + return c && r.UNIT_MEASURE === 'IX' + ? { id: `oecd:cpi:${r.REF_AREA}:${c.name}`, region: r.REF_AREA, code: r.EXPENDITURE } + : null; + }, + (code) => ({ title: OECD_CPI_CATEGORIES[code].title, unit: 'Index 2015=100' }) + ); + log(`oecd cpi: ${n} obs`); + total += n; + } catch (error) { + note(`cpi: ${error instanceof Error ? error.message : String(error)}`); + } + if (want('irlt')) + try { + const n = await pull( + `OECD.SDD.STES,DSD_STES@DF_FINMARK,/${IRLT_AREAS.join('+')}.M.IRLT......?startPeriod=1950-01`, + (r) => + r.MEASURE === 'IRLT' + ? { + id: `oecd:irlt:${region(r.REF_AREA)}`, + region: region(r.REF_AREA), + code: r.REF_AREA + } + : null, + () => ({ title: 'Long-term interest rate (10-year government bonds)', unit: '%' }) + ); + log(`oecd irlt: ${n} obs`); + total += n; + } catch (error) { + note(`irlt: ${error instanceof Error ? error.message : String(error)}`); + } + if (!total) throw new Error('nothing loaded'); + return total; + }); +} diff --git a/scripts/stats/statcan.ts b/scripts/stats/statcan.ts new file mode 100644 index 0000000..c7b0670 --- /dev/null +++ b/scripts/stats/statcan.ts @@ -0,0 +1,57 @@ +// Statistics Canada Web Data Service (no key): CPI table 18-10-0004-01 (2002 = 100, monthly, +// not seasonally adjusted) by vector. Normalised only: `statcan:cpi:`. +import { getJson, log, run, writeSeries, type ObsRow, type SeriesRow } from './common'; + +export const STATCAN_CPI: { vector: number; name: string; title: string }[] = [ + { vector: 41690973, name: 'total', title: 'Canada consumer price index, all items' }, + { vector: 41690974, name: 'food', title: 'Canada consumer price index, food' }, + { vector: 41691050, name: 'housing', title: 'Canada consumer price index, shelter' }, + { vector: 41691239, name: 'energy', title: 'Canada consumer price index, energy' }, + { + vector: 41691233, + name: 'core', + title: 'Canada consumer price index, all items excluding food and energy' + } +]; + +type Wds = { + status: string; + object?: { vectorId: number; vectorDataPoint: { refPer: string; value: number | null }[] }; +}[]; + +export async function statcan(opts: { job: string }) { + await run('statcan', opts.job, async (note) => { + const ids = STATCAN_CPI.map((s) => `%22${s.vector}%22`).join(','); + const body = await getJson( + `https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorByReferencePeriodRange?vectorIds=${ids}&startRefPeriod=1900-01-01&endReferencePeriod=2100-12-01` + ); + const series: SeriesRow[] = []; + const obs: ObsRow[] = []; + for (const s of STATCAN_CPI) { + const points = body.find((b) => b.object?.vectorId === s.vector)?.object?.vectorDataPoint; + if (!points?.length) { + note(`v${s.vector}: no data`); + continue; + } + const id = `statcan:cpi:${s.name}`; + series.push({ + series_id: id, + source_id: 'statcan', + source_code: `18-10-0004-01/v${s.vector}`, + title: s.title, + unit: 'Index 2002=100', + freq: 'M', + region_code: 'CAN', + theme: 'economy', + meta: { url: 'https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1810000401' } + }); + for (const p of points) + if (p.value != null && /^\d{4}-\d{2}-01$/.test(p.refPer)) + obs.push({ series_id: id, period: p.refPer, value: p.value, status: null }); + } + if (!obs.length) throw new Error('no StatCan observations'); + await writeSeries(series, obs); + log(`statcan: ${obs.length} obs in ${series.length} series`); + return obs.length; + }); +} diff --git a/scripts/stats/worldbank.ts b/scripts/stats/worldbank.ts index b9f875a..452f6c9 100644 --- a/scripts/stats/worldbank.ts +++ b/scripts/stats/worldbank.ts @@ -27,6 +27,7 @@ export const WB_INDICATORS: { id: string; title: string; unit: string; theme: st { id: 'NY.GDP.MKTP.CD', title: 'GDP', unit: 'current USD', theme: 'countries' }, { id: 'NY.GDP.MKTP.KD.ZG', title: 'GDP growth', unit: '%', theme: 'countries' }, { id: 'FP.CPI.TOTL.ZG', title: 'Inflation, consumer prices', unit: '%', theme: 'countries' }, + { id: 'FP.CPI.TOTL', title: 'Consumer price index', unit: 'Index 2010=100', theme: 'economy' }, { id: 'SP.POP.TOTL', title: 'Population', unit: 'people', theme: 'countries' }, { id: 'SM.POP.NETM', title: 'Net migration', unit: 'people', theme: 'countries' }, { diff --git a/src/lib/stats/CountryPicker.svelte b/src/lib/stats/CountryPicker.svelte new file mode 100644 index 0000000..febb921 --- /dev/null +++ b/src/lib/stats/CountryPicker.svelte @@ -0,0 +1,96 @@ + + +
+
+ core + {#each CORE7 as c (c)} + + {/each} + +
+
+ add + {#each EXTRA as c (c)} + + {/each} + {#if !isCore} + + {/if} +
+
+ + diff --git a/src/lib/stats/countries.ts b/src/lib/stats/countries.ts index 53a0f93..866db71 100644 --- a/src/lib/stats/countries.ts +++ b/src/lib/stats/countries.ts @@ -13,6 +13,7 @@ export const COUNTRY_PAGES = [ 'BEL', 'GBR', 'JPN', + 'CAN', 'IND', 'BRA', 'RUS', @@ -34,6 +35,7 @@ export const FLAG: Record = { BEL: '🇧🇪', GBR: '🇬🇧', JPN: '🇯🇵', + CAN: '🇨🇦', IND: '🇮🇳', BRA: '🇧🇷', RUS: '🇷🇺', @@ -52,6 +54,7 @@ export const COUNTRY_COLOR: Record = { BEL: '#a16207', GBR: '#a855f7', JPN: '#ec4899', + CAN: '#dc2626', IND: '#10b981', BRA: '#84cc16', RUS: '#0d9488', diff --git a/src/lib/stats/econ.test.ts b/src/lib/stats/econ.test.ts new file mode 100644 index 0000000..bd55f3d --- /dev/null +++ b/src/lib/stats/econ.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it } from 'vitest'; +import { CORE7, CPI_BY_CATEGORY, CPI_TOTAL, rebase, yearMean, yoyByDate } from './econ.js'; + +describe('econ helpers', () => { + const pts: [string, number][] = [ + ['2014-12-01', 99], + ['2015-01-01', 100], + ['2015-07-01', 102], + ['2016-01-01', 103], + ['2016-07-01', 104.04] + ]; + it('averages a calendar year', () => expect(yearMean(pts, 2015)).toBe(101)); + it('rebases to the base-year mean', () => + expect(rebase(pts, 2015)?.[3]).toEqual(['2016-01-01', 101.98])); + it('returns null when the base year is missing', () => expect(rebase(pts, 2010)).toBeNull()); + it('computes year on year by date and skips gaps', () => + expect(yoyByDate(pts)).toEqual([ + ['2016-01-01', 3], + ['2016-07-01', 2] + ])); + it('covers every core country for CPI', () => { + for (const c of CORE7) { + expect(CPI_TOTAL[c]).toBeDefined(); + expect(CPI_BY_CATEGORY[c]?.ids.total).toBeDefined(); + } + }); +}); diff --git a/src/lib/stats/econ.ts b/src/lib/stats/econ.ts new file mode 100644 index 0000000..31201da --- /dev/null +++ b/src/lib/stats/econ.ts @@ -0,0 +1,151 @@ +// Country sets and series maps for /stats/economy. The core set (CAN, KOR, CHN, JPN, USA, NLD, +// CUW) is the default on every multi-country chart; the extra countries can be added, and the +// EU aggregate is kept as a comparison line. Every id points at a stored series; nothing here is +// estimated, gaps stay gaps. +import type { StatPoint } from './chart.js'; + +export const CORE7 = ['CAN', 'KOR', 'CHN', 'JPN', 'USA', 'NLD', 'CUW'] as const; +export const EXTRA = ['DEU', 'FRA', 'GBR', 'BEL', 'IND', 'BRA', 'RUS'] as const; +export const ECON_COUNTRIES = [...CORE7, ...EXTRA] as const; +export type EconCountry = (typeof ECON_COUNTRIES)[number]; +/** Comparison aggregates: EU 27 by default; the euro area where the data is per currency area. */ +export type Aggregate = 'EU27' | 'EA'; + +/** All-items CPI, index level, best monthly source per region (national office first). */ +export const CPI_TOTAL: Record = { + // CBS CPI as carried by the BIS long series (1965→). CBS 83131NED (2015 = 100) stops in + // December 2025 and 86141NED (2025 = 100) starts in December 2009, so neither covers the span. + NLD: { id: 'bis:cpi:NL', source: 'bis', base: '2010 = 100' }, + USA: { id: 'fred:CPIAUCSL', source: 'fred', base: '1982–84 = 100' }, + CAN: { id: 'statcan:cpi:total', source: 'statcan', base: '2002 = 100' }, + JPN: { id: 'bis:cpi:JP', source: 'bis', base: '2010 = 100' }, + KOR: { id: 'bis:cpi:KR', source: 'bis', base: '2010 = 100' }, + CHN: { id: 'bis:cpi:CN', source: 'bis', base: '2010 = 100' }, + CUW: { id: 'cw:cpi:index', source: 'cbscw', base: '2023 = 100' }, + DEU: { id: 'bis:cpi:DE', source: 'bis', base: '2010 = 100' }, + FRA: { id: 'bis:cpi:FR', source: 'bis', base: '2010 = 100' }, + GBR: { id: 'bis:cpi:GB', source: 'bis', base: '2010 = 100' }, + BEL: { id: 'bis:cpi:BE', source: 'bis', base: '2010 = 100' }, + IND: { id: 'bis:cpi:IN', source: 'bis', base: '2010 = 100' }, + BRA: { id: 'bis:cpi:BR', source: 'bis', base: '2010 = 100' }, + RUS: { id: 'bis:cpi:RU', source: 'bis', base: '2010 = 100' }, + EU27: { + id: 'eurostat:prc_hicp_minr:I15.TOTAL.EU27_2020', + source: 'eurostat', + base: '2015 = 100' + }, + EA: { id: 'eurostat:prc_hicp_minr:I15.TOTAL.EA20', source: 'eurostat', base: '2015 = 100' } +}; + +export const CPI_CATS = ['total', 'food', 'housing', 'energy', 'core'] as const; +export type CpiCat = (typeof CPI_CATS)[number]; +export const CPI_CAT_LABEL: Record = { + total: 'All items', + food: 'Food', + housing: 'Housing', + energy: 'Energy', + core: 'Core (excl. food and energy)' +}; +const hicp = (g: string): Record => ({ + total: `eurostat:prc_hicp_minr:I15.TOTAL.${g}`, + food: `eurostat:prc_hicp_minr:I15.CP01.${g}`, + housing: `eurostat:prc_hicp_minr:I15.CP04.${g}`, + energy: `eurostat:prc_hicp_minr:I15.NRG.${g}`, + core: `eurostat:prc_hicp_minr:I15.TOT_X_NRG_FOOD.${g}` +}); +const oecd = (c: string, cats: CpiCat[]) => + Object.fromEntries(cats.map((k) => [k, `oecd:cpi:${c}:${k}`])) as Partial>; + +/** CPI by main category where the source publishes it. */ +export const CPI_BY_CATEGORY: Record< + string, + { ids: Partial>; source: string; note: string } +> = { + NLD: { ids: hicp('NL'), source: 'eurostat', note: 'HICP (Eurostat), 2015 = 100' }, + USA: { + ids: { + total: 'fred:CPIAUCSL', + food: 'fred:CPIUFDSL', + housing: 'fred:CPIHOSSL', + energy: 'fred:CPIENGSL', + core: 'fred:CPILFESL' + }, + source: 'fred', + note: 'BLS CPI-U, seasonally adjusted, via FRED' + }, + CAN: { + ids: Object.fromEntries(CPI_CATS.map((k) => [k, `statcan:cpi:${k}`])), + source: 'statcan', + note: 'Statistics Canada 18-10-0004-01; housing = shelter' + }, + KOR: { + ids: oecd('KOR', [...CPI_CATS]), + source: 'oecd', + note: 'OECD prices database' + }, + JPN: { + ids: oecd('JPN', [...CPI_CATS]), + source: 'oecd', + note: 'OECD prices database; the OECD series for Japan stop in June 2021' + }, + CHN: { + ids: oecd('CHN', ['total', 'food']), + source: 'oecd', + note: 'OECD prices database: all items and food only (other NBS categories are manual)' + }, + CUW: { + ids: { total: 'cw:cpi:index' }, + source: 'cbscw', + note: 'CBS Curaçao: all items only' + }, + EU27: { ids: hicp('EU27_2020'), source: 'eurostat', note: 'HICP (Eurostat), 2015 = 100' }, + EA: { ids: hicp('EA20'), source: 'eurostat', note: 'HICP (Eurostat), 2015 = 100' } +}; + +/** BIS policy rate area per country (euro members → ECB); missing = no BIS series. */ +export const POLICY_RATE: Record = { + CAN: 'CA', + KOR: 'KR', + CHN: 'CN', + JPN: 'JP', + USA: 'US', + NLD: 'XM', + GBR: 'GB', + EA: 'XM' +}; + +/** Annual macro indicators: IMF WEO first, then CBS Curaçao for Curaçao, then the World Bank. */ +export const ANNUAL = { + growth: { imf: 'NGDP_RPCH', wb: 'NY.GDP.MKTP.KD.ZG', cw: 'cw:gdp:real_growth_pct' }, + unemployment: { + imf: 'LUR', + wb: 'SL.UEM.TOTL.ZS', + cw: 'cw:lfs:unemployment_rate_pct:total' + }, + debt: { imf: 'GGXWDG_NGDP', wb: 'GC.DOD.TOTL.GD.ZS', cw: null }, + trade: { imf: null, wb: 'NE.TRD.GNFS.ZS', cw: null } +} as const; + +/** Mean of the points in a calendar year, or null when the year has no observations. */ +export function yearMean(points: StatPoint[], year: number): number | null { + const vals = points.filter(([d]) => d.startsWith(`${year}-`)).map(([, v]) => v); + return vals.length ? vals.reduce((a, b) => a + b, 0) / vals.length : null; +} + +/** Rebase an index to (base year average) = 100. Null when the base year is not covered. */ +export function rebase(points: StatPoint[], year: number): StatPoint[] | null { + const base = yearMean(points, year); + if (!base) return null; + return points.map(([d, v]) => [d, Math.round((v / base) * 10000) / 100]); +} + +/** Year-on-year % change by date (same month a year earlier); missing months stay missing. */ +export function yoyByDate(points: StatPoint[]): StatPoint[] { + const map = new Map(points); + const out: StatPoint[] = []; + for (const [d, v] of points) { + const prev = map.get(`${Number(d.slice(0, 4)) - 1}${d.slice(4)}`); + if (prev) out.push([d, Math.round(((v - prev) / prev) * 10000) / 100]); + } + return out; +} diff --git a/src/lib/stats/regions.ts b/src/lib/stats/regions.ts index e336d41..6060218 100644 --- a/src/lib/stats/regions.ts +++ b/src/lib/stats/regions.ts @@ -19,6 +19,7 @@ export const COUNTRY: Record = { CN: 'China', GB: 'United Kingdom', JP: 'Japan', + CA: 'Canada', NLD: 'Netherlands', USA: 'United States', KOR: 'South Korea', @@ -31,6 +32,7 @@ export const COUNTRY: Record = { BEL: 'Belgium', GBR: 'United Kingdom', JPN: 'Japan', + CAN: 'Canada', IND: 'India', BRA: 'Brazil', RUS: 'Russia', diff --git a/src/routes/stats/economy/+page.server.ts b/src/routes/stats/economy/+page.server.ts index c229122..6a879f3 100644 --- a/src/routes/stats/economy/+page.server.ts +++ b/src/routes/stats/economy/+page.server.ts @@ -1,30 +1,49 @@ import { cached, loadSeries } from '#lib/server/stats.js'; +import { + ANNUAL, + CPI_BY_CATEGORY, + CPI_TOTAL, + ECON_COUNTRIES, + POLICY_RATE +} from '#lib/stats/econ.js'; import type { PageServerLoad } from './$types'; const EU = ['NL', 'DE', 'FR', 'BE', 'IT', 'ES', 'PL', 'SE', 'DK', 'AT', 'EU27_2020', 'EA20']; const YIELDS = ['NL', 'DE', 'FR', 'IT', 'ES', 'BE', 'AT', 'EA']; +const REGIONS = [...ECON_COUNTRIES, 'EU27', 'EA']; export const load: PageServerLoad = async () => ({ series: await cached('stats:economy', () => loadSeries([ - ...['US', 'XM', 'KR', 'CN', 'GB', 'JP'].map((a) => ({ + ...[...new Set(Object.values(POLICY_RATE))].map((a) => ({ id: `bis:M:${a}`, from: '1990-01-01' })), ...EU.map((g) => `eurostat:prc_hicp_minr:RCH_A.TOTAL.${g}`), - { id: 'fred:CPIAUCSL', from: '1989-01-01' }, + // CPI index levels (all items, then the categories) + ...[ + ...new Set([ + ...Object.values(CPI_TOTAL).map((c) => c.id), + ...Object.values(CPI_BY_CATEGORY).flatMap((c) => Object.values(c.ids)) + ]) + ].map((id) => ({ id, from: '1960-01-01' })), + ...REGIONS.map((c) => ({ id: `oecd:irlt:${c}`, from: '1980-01-01' })), ...YIELDS.map((g) => `eurostat:irt_lt_mcby_m:MCBY.${g}`), { id: 'fred:DGS10', from: '1990-01-01', resample: 'M' }, { id: 'fred:DGS2', from: '1990-01-01', resample: 'M' }, - 'fred:IRLTLT01KRM156N', ...['NL', 'KR', 'US', 'CN'].map((c) => ({ id: `fred:SPASTT01${c}M661N`, from: '1990-01-01' })), { id: 'fred:SP500', resample: 'W' }, { id: 'fred:NASDAQCOM', from: '1990-01-01', resample: 'W' }, - ...EU.map((g) => `eurostat:nama_10_gdp:CLV_PCH_PRE.B1GQ.${g}`), - ...EU.map((g) => `eurostat:nama_10_gdp:CP_MEUR.B1GQ.${g}`), + ...Object.values(ANNUAL).flatMap((ind) => + REGIONS.flatMap((c) => [ + ...(ind.imf ? [{ id: `imf:${ind.imf}:${c}`, from: '1980-01-01' }] : []), + { id: `wb:${ind.wb}:${c}`, from: '1980-01-01' } + ]) + ), + ...Object.values(ANNUAL).flatMap((ind) => (ind.cw ? [ind.cw] : [])), 'fred:GDPC1', 'fred:UNRATE', ...EU.flatMap((g) => [ diff --git a/src/routes/stats/economy/+page.svelte b/src/routes/stats/economy/+page.svelte index 522ef05..35d0279 100644 --- a/src/routes/stats/economy/+page.svelte +++ b/src/routes/stats/economy/+page.svelte @@ -1,25 +1,159 @@ economy · stats · Stan Runge @@ -73,9 +207,10 @@ stats / economy

Economy

- Central bank rates, inflation, 10-year bond yields and spreads, stock markets, growth, public - finances and migration. The Netherlands next to Europe, the US and South Korea, refreshed daily - from open sources. + Consumer prices, central bank rates, bond yields, growth, jobs, debt and trade for Canada, South + Korea, China, Japan, the US, the Netherlands and Curaçao, with the EU as a comparison line. Add + other countries with the picker. Refreshed from open sources (CBS, BLS, Statistics Canada, CBS + Curaçao, BIS, OECD, IMF, World Bank, Eurostat).

@@ -95,11 +230,18 @@ color="#eab308" /> + - -

Rates and inflation

+ + +

Prices and rates

+ + lineOption(t, cpiLines.lines, { unit: cpiMode === 'level' ? 'index' : '%', zoom: true })} + > +
+ + + index level + year on year % + + + {#if cpiMode === 'level'} + + {/if} +
+ + + lineOption(t, catLines, { unit: cpiMode === 'level' ? 'index' : '%', zoom: true })} + > +
+ + uses the index / year-on-year switch and base year above +
+ - lineOption( - t, - RATES.map(([c, name, color]) => ({ name, color, kind: 'step', points: pts(`bis:M:${c}`) })), - { unit: '%', zoom: true } - )} + note={`Euro members follow the ECB. ${missing(rateLines.gaps)}${rateLines.gaps.includes('CUW') ? ' Curaçao: the CBCS pegs the guilder to the US dollar and is not in the BIS policy rate set.' : ''}`} + option={(t) => lineOption(t, rateLines.lines, { unit: '%', zoom: true })} /> - lineOption( - t, - [ - { name: 'Netherlands', points: hicp('NL'), color: '#f97316' }, - { name: 'Euro area', points: hicp('EA20'), color: '#10b981' }, - { name: 'Germany', points: hicp('DE'), color: '#64748b' }, - { name: 'United States', points: usCpi, color: '#3b82f6' } - ], - { unit: '%', zoom: true } - )} + note={missing(yieldLines.gaps)} + option={(t) => lineOption(t, yieldLines.lines, { unit: '%', zoom: true })} />
-

Bond markets

+

Growth, jobs and debt

- lineOption( - t, - [ - { name: 'Netherlands', points: y('NL'), color: '#f97316' }, - { name: 'Germany', points: y('DE'), color: '#64748b' }, - { name: 'France', points: y('FR'), color: '#3b82f6' }, - { name: 'Italy', points: y('IT'), color: '#10b981' }, - { name: 'Spain', points: y('ES'), color: '#eab308' }, - { name: 'United States', points: us10, color: '#a855f7' }, - { name: 'South Korea', points: pts('fred:IRLTLT01KRM156N'), color: '#e11d48' } - ], - { unit: '%', zoom: true } - )} + note={`IMF World Economic Outlook; CBS Curaçao (real growth, 2011→) for Curaçao. ${missing(growth.gaps)}`} + option={(t) => lineOption(t, growth.lines, { unit: '%', zoom: true })} /> + lineOption(t, unemployment.lines, { unit: '%', zoom: true })} + /> + lineOption(t, govDebt.lines, { unit: '% of GDP', zoom: true })} + /> + lineOption(t, trade.lines, { unit: '% of GDP', zoom: true })} + /> +
+ +

Bond and stock markets

+
-

Growth and jobs

+

United States

- - lineOption( - t, - [ - { - name: 'Netherlands', - points: pts('eurostat:nama_10_gdp:CLV_PCH_PRE.B1GQ.NL'), - color: '#f97316', - kind: 'bar' - }, - { - name: 'EU 27', - points: pts('eurostat:nama_10_gdp:CLV_PCH_PRE.B1GQ.EU27_2020'), - color: '#3b82f6' - }, - { - name: 'Germany', - points: pts('eurostat:nama_10_gdp:CLV_PCH_PRE.B1GQ.DE'), - color: '#64748b' - } - ], - { unit: '%' } - )} - />
-

Public finances and migration

+

Europe: public finances and migration

- Policy rates: BIS central bank policy rates (month end). Yields for the euro countries are - Eurostat's EMU convergence-criterion series. + Policy rates: BIS central bank policy rates (month end). Spreads use Eurostat's EMU + convergence-criterion yields. The country picker drives the charts in the first two sections; the + European and US sections keep their own fixed sets.

+ +