houses: nightly Kamernet + Vesteda scraper, dedupe by address, owner-only /trackers/houses
All checks were successful
Deploy production / deploy (push) Successful in 2m44s
All checks were successful
Deploy production / deploy (push) Successful in 2m44s
- scripts/house-scrape.ts: polite nightly scrape (honest UA, 6-10 s apart, stops a source on 403/challenge), ingest + price history + removals, ntfy alerts on a separate topic (max 5 + summary, first run seeds)
- address_key + duplicate_of: copies on another source link to one listing, manual wins
- /projects/house-tracker -> /trackers/houses (308), owner only (404 otherwise); filters for min m2, rooms, source, Gouda region default; notes, add-by-hand form, total rent, also-on links
- scripts/house-db.sh: run the house-listing CLI as the limited house_cli role
- deploy/systemd/house-scrape.{service,timer}
This commit is contained in:
@@ -1,7 +1,38 @@
|
||||
# House tracker
|
||||
|
||||
## Nightly scrape on stan-vm (2026-10)
|
||||
|
||||
`scripts/house-scrape.ts` runs once a night from a systemd user timer
|
||||
(`deploy/systemd/house-scrape.{service,timer}`, 03:40 Amsterdam + up to 20 min jitter) and
|
||||
writes `house.listings` directly with the limited `house_scraper` role
|
||||
(`~/.config/house-scrape.env`, mode 600). One request at a time, 6–10 s apart, honest
|
||||
User-Agent, robots.txt-allowed paths only. A 403/429/challenge stops that source for the
|
||||
night and is logged as `challenged`; nothing is retried or worked around.
|
||||
|
||||
| source | how | status |
|
||||
| ---------------------- | ------------------------------------------------------------- | ------------------------------------ |
|
||||
| Kamernet | `/huren/huurwoningen-<city>` pages, `__NEXT_DATA__` JSON | scraped (10 cities around Gouda) |
|
||||
| Vesteda | `POST /api/units/search/facet` (what their site calls), 20 km | scraped; rented units count as gone |
|
||||
| Pararius | Cloudflare 403 from the Hetzner IP | not scraped here; home worker below |
|
||||
| huurwoningen.nl | same Cloudflare 403 (Pararius group) | skipped |
|
||||
| Funda | bot-check page ("Je bent bijna op de pagina…") | skipped |
|
||||
| Huurstunt | Cloudflare challenge | skipped |
|
||||
| Mijndak Midden-Holland | OutSystems app behind a login, no public listing feed | skipped; use their own e-mail alerts |
|
||||
|
||||
Same home on two sources: `address_key` (street + number + city, normalised, see
|
||||
`src/lib/house-tracker/address.ts`) links the copy to the original via `duplicate_of`; a
|
||||
manual entry wins. The page shows one row with "also on …" links. Kamernet never shows house
|
||||
numbers, so its listings are never merged. Alerts: new non-copy listings matching an alert
|
||||
rule go to `HOUSE_NTFY_URL` (its own topic, max 5 per night plus one summary); the first run
|
||||
of a source only seeds.
|
||||
|
||||
Manual listings: `scripts/house-listing.ts` (CLI) or the form on the page. Against prod use
|
||||
`scripts/house-db.sh <command> …`, which runs the CLI as `house_cli` (SELECT/INSERT/UPDATE/
|
||||
DELETE on house.listings, SELECT/INSERT on house.price_history, nothing else) from
|
||||
`~/.config/house-cli.env`.
|
||||
|
||||
Personal tool: collects Dutch housing listings into Postgres, shows them on
|
||||
`/projects/house-tracker` (admin only) and sends an alert when a new listing
|
||||
`/trackers/houses` (owner only, 404 for everyone else; the old URL redirects) and sends an alert when a new listing
|
||||
matches a rule. Only Pararius for now; other sources plug in the same way.
|
||||
|
||||
## How it fits together
|
||||
|
||||
Reference in New Issue
Block a user