house-scrape: re-check huispedia, huislijn, huurportaal and rentola rows and mark clearly gone ones removed
All checks were successful
Deploy production / deploy (push) Successful in 1m15s
All checks were successful
Deploy production / deploy (push) Successful in 1m15s
This commit is contained in:
@@ -19,6 +19,24 @@ night and is logged as `challenged`; nothing is retried or worked around.
|
||||
| Huurstunt | Cloudflare challenge | skipped |
|
||||
| Mijndak Midden-Holland | OutSystems app behind a login, no public listing feed | skipped; use their own e-mail alerts |
|
||||
|
||||
**Re-check of imported rows** (`src/lib/house-tracker/recheck.ts`). Rows added by the CLI or
|
||||
one-off backfills from huispedia, huislijn, huurportaal and rentola are not in any search
|
||||
feed, so after the sources the scrape fetches each row's own page (all four allow detail
|
||||
pages in robots.txt), up to 400 per site, oldest `last_seen_at` first, within
|
||||
`HOUSE_RECHECK_MINUTES` (default 8). Each site has its own one-at-a-time 6–10 s queue.
|
||||
Alive sets `last_seen_at`; gone sets `removed_at` and needs a clear signal:
|
||||
|
||||
| site | gone when |
|
||||
| -------------------- | ------------------------------------------------------------------------------- |
|
||||
| huurportaal, rentola | redirect to a search page with `?inactive_property_id=`, or 404/410 |
|
||||
| huislijn | redirect to the province overview (id gone from the URL), 404/410, woningruil |
|
||||
| huispedia | 404/410 ("niet meer beschikbaar"), or Status verhuurd / verkocht / niet te koop |
|
||||
|
||||
Errors, timeouts, off-site redirects and unfamiliar pages are `unknown` and change nothing; a
|
||||
403/429/503 or challenge stops that site for the night. If more than half of a site's checks
|
||||
in one night (min. 5) say gone, none are applied and the run is logged as `suspicious`.
|
||||
Try it with `--dry-run --source recheck --recheck-limit 5`.
|
||||
|
||||
Same home on two sources: `address_key` (street + number + city, normalised, see
|
||||
`src/lib/house-tracker/address.ts`) links the copy to the original via `duplicate_of`; a
|
||||
manual entry wins. The page shows one row with "also on …" links. Kamernet never shows house
|
||||
|
||||
Reference in New Issue
Block a user