house-scrape: re-check huispedia, huislijn, huurportaal and rentola rows and mark clearly gone ones removed
All checks were successful
Deploy production / deploy (push) Successful in 1m15s

This commit is contained in:
2026-10-25 23:14:42 +00:00
parent 241b90d2f9
commit fa3aa25f0b
5 changed files with 398 additions and 5 deletions

View File

@@ -19,6 +19,24 @@ night and is logged as `challenged`; nothing is retried or worked around.
| Huurstunt | Cloudflare challenge | skipped |
| Mijndak Midden-Holland | OutSystems app behind a login, no public listing feed | skipped; use their own e-mail alerts |
**Re-check of imported rows** (`src/lib/house-tracker/recheck.ts`). Rows added by the CLI or
one-off backfills from huispedia, huislijn, huurportaal and rentola are not in any search
feed, so after the sources the scrape fetches each row's own page (all four allow detail
pages in robots.txt), up to 400 per site, oldest `last_seen_at` first, within
`HOUSE_RECHECK_MINUTES` (default 8). Each site has its own one-at-a-time 6–10 s queue.
Alive sets `last_seen_at`; gone sets `removed_at` and needs a clear signal:
| site | gone when |
| -------------------- | ------------------------------------------------------------------------------- |
| huurportaal, rentola | redirect to a search page with `?inactive_property_id=`, or 404/410 |
| huislijn | redirect to the province overview (id gone from the URL), 404/410, woningruil |
| huispedia | 404/410 ("niet meer beschikbaar"), or Status verhuurd / verkocht / niet te koop |
Errors, timeouts, off-site redirects and unfamiliar pages are `unknown` and change nothing; a
403/429/503 or challenge stops that site for the night. If more than half of a site's checks
in one night (min. 5) say gone, none are applied and the run is logged as `suspicious`.
Try it with `--dry-run --source recheck --recheck-limit 5`.
Same home on two sources: `address_key` (street + number + city, normalised, see
`src/lib/house-tracker/address.ts`) links the copy to the original via `duplicate_of`; a
manual entry wins. The page shows one row with "also on …" links. Kamernet never shows house