houses: nightly Kamernet + Vesteda scraper, dedupe by address, owner-only /trackers/houses
All checks were successful
Deploy production / deploy (push) Successful in 2m44s

- scripts/house-scrape.ts: polite nightly scrape (honest UA, 6-10 s apart, stops a source on 403/challenge), ingest + price history + removals, ntfy alerts on a separate topic (max 5 + summary, first run seeds)
- address_key + duplicate_of: copies on another source link to one listing, manual wins
- /projects/house-tracker -> /trackers/houses (308), owner only (404 otherwise); filters for min m2, rooms, source, Gouda region default; notes, add-by-hand form, total rent, also-on links
- scripts/house-db.sh: run the house-listing CLI as the limited house_cli role
- deploy/systemd/house-scrape.{service,timer}
This commit is contained in:
2026-10-24 14:07:33 +00:00
parent ea7d64a556
commit b51fef14fd
24 changed files with 18142 additions and 35 deletions

View File

@@ -1,7 +1,38 @@
# House tracker
## Nightly scrape on stan-vm (2026-10)
`scripts/house-scrape.ts` runs once a night from a systemd user timer
(`deploy/systemd/house-scrape.{service,timer}`, 03:40 Amsterdam + up to 20 min jitter) and
writes `house.listings` directly with the limited `house_scraper` role
(`~/.config/house-scrape.env`, mode 600). One request at a time, 6–10 s apart, honest
User-Agent, robots.txt-allowed paths only. A 403/429/challenge stops that source for the
night and is logged as `challenged`; nothing is retried or worked around.
| source | how | status |
| ---------------------- | ------------------------------------------------------------- | ------------------------------------ |
| Kamernet | `/huren/huurwoningen-<city>` pages, `__NEXT_DATA__` JSON | scraped (10 cities around Gouda) |
| Vesteda | `POST /api/units/search/facet` (what their site calls), 20 km | scraped; rented units count as gone |
| Pararius | Cloudflare 403 from the Hetzner IP | not scraped here; home worker below |
| huurwoningen.nl | same Cloudflare 403 (Pararius group) | skipped |
| Funda | bot-check page ("Je bent bijna op de pagina…") | skipped |
| Huurstunt | Cloudflare challenge | skipped |
| Mijndak Midden-Holland | OutSystems app behind a login, no public listing feed | skipped; use their own e-mail alerts |
Same home on two sources: `address_key` (street + number + city, normalised, see
`src/lib/house-tracker/address.ts`) links the copy to the original via `duplicate_of`; a
manual entry wins. The page shows one row with "also on …" links. Kamernet never shows house
numbers, so its listings are never merged. Alerts: new non-copy listings matching an alert
rule go to `HOUSE_NTFY_URL` (its own topic, max 5 per night plus one summary); the first run
of a source only seeds.
Manual listings: `scripts/house-listing.ts` (CLI) or the form on the page. Against prod use
`scripts/house-db.sh <command> …`, which runs the CLI as `house_cli` (SELECT/INSERT/UPDATE/
DELETE on house.listings, SELECT/INSERT on house.price_history, nothing else) from
`~/.config/house-cli.env`.
Personal tool: collects Dutch housing listings into Postgres, shows them on
`/projects/house-tracker` (admin only) and sends an alert when a new listing
`/trackers/houses` (owner only, 404 for everyone else; the old URL redirects) and sends an alert when a new listing
matches a rule. Only Pararius for now; other sources plug in the same way.
## How it fits together