stats: fix role-name quoting in the stats migration grants; add deploy/stats-collector (systemd user timers for CBS, Eurostat, FRED, BIS)
All checks were successful
Deploy production / deploy (push) Successful in 1m15s
All checks were successful
Deploy production / deploy (push) Successful in 1m15s
This commit is contained in:
28
deploy/stats-collector/README.md
Normal file
28
deploy/stats-collector/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# stats collector
|
||||
|
||||
Host-side collector for the public [/stats](https://stanrunge.dev/stats) section. It pulls open
|
||||
statistics (no API keys) into the `stats` schema with `scripts/stats-fetch.ts`:
|
||||
|
||||
| source | timer | schedule (Europe/Amsterdam) | what |
|
||||
|---|---|---|---|
|
||||
| `cbs` | `stats-cbs.timer` | nightly 05:40 | CBS StatLine OData v4: checks every table's `Modified`, re-pulls changed ones, flags discontinued tables (base-year swaps) |
|
||||
| `eurostat` | `stats-eurostat.timer` | Mondays 06:20 | Eurostat JSON-stat datasets (energy prices, yields, HICP, government finance…) |
|
||||
| `fred` | `stats-fred.timer` | daily 07:20 | FRED graph CSV (Brent, TTF, Fed, Treasuries, US energy prices, indices) |
|
||||
| `bis` | `stats-bis.timer` | daily 07:50 | BIS central bank policy rates, monthly + daily |
|
||||
|
||||
Each timer starts `stats-collector@<source>.service`, which runs
|
||||
`~/.local/bin/stats-collector-run.sh <source>`: it updates a detached checkout of `origin/main`
|
||||
in `~/dev/stats-runner`, reads `DB_URL` (app role) from the running app container, and runs the
|
||||
fetch. Every job is logged in `stats.fetch_run` (shown as "data freshness" on /stats).
|
||||
|
||||
Politeness: one shared User-Agent, a per-host gap between requests (CBS 250 ms ≈ 4 req/s,
|
||||
OECD 3.1 s ≈ 19/min, FRED/BIS 1 s), retries with backoff on 429/5xx.
|
||||
|
||||
```sh
|
||||
bash deploy/stats-collector/install.sh # install units + enable timers
|
||||
systemctl --user start stats-collector@fred.service # run one source now
|
||||
# long backfills: detached, survive SSH drops; logs in journalctl --user -u stats-backfill-cbs
|
||||
systemd-run --user --unit=stats-backfill-cbs ~/.local/bin/stats-collector-run.sh cbs --backfill
|
||||
```
|
||||
|
||||
Never touches the `energy` schema or ned-collector.
|
||||
Reference in New Issue
Block a user