Stan Runge
6e528da403
scripts/house-listing.ts (bun run house add|update|list|remove|restore) upserts house.listings on (source, source_id) derived from the listing URL or address, derives city_slug, records price_history, and accepts flags or JSON.
205 lines
9.6 KiB
Markdown
205 lines
9.6 KiB
Markdown
# House tracker
|
||
|
||
Personal tool: collects Dutch housing listings into Postgres, shows them on
|
||
`/projects/house-tracker` (admin only) and sends an alert when a new listing
|
||
matches a rule. Only Pararius for now; other sources plug in the same way.
|
||
|
||
## How it fits together
|
||
|
||
```
|
||
home box (residential IP) stanrunge.dev (Hetzner)
|
||
┌──────────────────────────┐ POST sync ┌──────────────────────────────┐
|
||
│ scripts/house-worker.ts │ ──────────────▶ │ /projects/house-tracker/api/ │
|
||
│ polls pararius.nl │ POST details │ worker/{sync,details,report}│
|
||
│ parses HTML │ ◀────────────── │ stores, dedups, alerts │
|
||
└──────────────────────────┘ needDetails └──────────────┬───────────────┘
|
||
▼
|
||
Discord webhook / ntfy push
|
||
```
|
||
|
||
The server never talks to the listing sites. Pararius sits behind Cloudflare,
|
||
which challenges datacenter IPs and plain HTTP clients, so the worker runs at
|
||
home where traffic looks like a normal visitor.
|
||
|
||
Two lanes:
|
||
|
||
- **fast**: one filtered search (default `huurwoningen/gouda/1300-1700`), page 1
|
||
only. Every 3–5 min on weekdays 07:00–18:00 Amsterdam time, every 30–60 min
|
||
otherwise. Listings near Gouda were all published in office hours on weekdays.
|
||
Alerts are sent from the search card alone, before the detail page is fetched.
|
||
- **slow**: full, paginated sweeps of rent and buy listings in Gouda, Rotterdam,
|
||
Den Haag, Utrecht, Amsterdam and Eindhoven every 12 h. Unfiltered city sweeps
|
||
also mark listings that disappeared, which gives "days listed" in the stats.
|
||
|
||
Safety rules in the worker: one request at a time, at least 8 s apart with
|
||
jitter, a daily budget (1500 by default, the slow lane may only use 60%), and
|
||
any challenge/403/429 stops everything and backs off 15 min → 30 → 60 … 6 h.
|
||
The first poll of any search only seeds the database, so you don't get 40
|
||
alerts for listings that were already online.
|
||
|
||
## Server setup
|
||
|
||
Add to the production env file (`/etc/personal-website/prod.env`):
|
||
|
||
```text
|
||
# openssl rand -hex 32
|
||
HOUSE_WORKER_TOKEN=
|
||
# one or both
|
||
HOUSE_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
|
||
HOUSE_NTFY_URL=https://ntfy.sh/some-long-random-topic
|
||
HOUSE_NTFY_TOKEN= # only for a protected ntfy topic
|
||
PUBLIC_MAPTILER_API_KEY= # already set for Animaldex
|
||
```
|
||
|
||
The `house` schema is part of the Drizzle migrations that run on every deploy. Use
|
||
"send test alert" on the page to check the alert channel.
|
||
|
||
If Cloudflare Bot Fight Mode / WAF challenges are on for stanrunge.dev, add a
|
||
WAF skip rule for `URI Path starts with /projects/house-tracker/api/worker/`,
|
||
otherwise the worker's POSTs get challenged too. The endpoints are protected by
|
||
the bearer token.
|
||
|
||
### Development behind Cloudflare Access
|
||
|
||
For `https://dev.stanrunge.dev`, keep Access enabled. In Cloudflare Zero Trust,
|
||
go to **Access controls → Service credentials → Service Tokens** and create a
|
||
token named `house-worker`. Save its Client ID and Client Secret. In the Access
|
||
application protecting the dev hostname, add a **Service Auth** policy with
|
||
**Include → Service Token → house-worker**. Keep the existing browser login
|
||
policy. See [Cloudflare's service token documentation](https://developers.cloudflare.com/cloudflare-one/access-controls/service-credentials/service-tokens/).
|
||
|
||
Set `HOUSE_WORKER_TOKEN` in the dev website's `.env` and restart its dev server.
|
||
On the home machine, copy `house.env.example` to `house.env`, fill in the same
|
||
worker token plus the Cloudflare Client ID and Client Secret, then run:
|
||
|
||
```sh
|
||
chmod 600 house.env
|
||
bun --env-file=house.env scripts/house-worker.ts --check-api
|
||
```
|
||
|
||
This makes one authenticated GET to the website's worker health endpoint. It
|
||
does not contact Pararius, sync listings or send alerts. The expected output is
|
||
`API connection OK`. Access credentials are sent only to the website API; the
|
||
client refuses redirects and stops immediately on an authentication failure.
|
||
An HTTP 503 can mean the website's worker token is missing; an HTTP 404 can mean
|
||
the running website does not yet include the health route.
|
||
|
||
## Home worker setup
|
||
|
||
Needs a machine that stays on (old laptop, Raspberry Pi, NAS) with Bun 1.4 or
|
||
newer; older versions cannot read this repository's lockfile. Use `bun upgrade`
|
||
if needed.
|
||
|
||
```sh
|
||
git clone https://git.stanrunge.dev/stan/website ~/house-worker
|
||
cd ~/house-worker && bun install --frozen-lockfile
|
||
cat > house.env <<'EOF'
|
||
HOUSE_API_URL=https://stanrunge.dev
|
||
HOUSE_WORKER_TOKEN=same-value-as-the-server
|
||
EOF
|
||
chmod 600 house.env
|
||
|
||
# 1. check parsing without sending anything
|
||
bun --env-file=house.env scripts/house-worker.ts --dry-run --once
|
||
# 2. one real poll
|
||
bun --env-file=house.env scripts/house-worker.ts --once
|
||
```
|
||
|
||
Then run it as a service, `/etc/systemd/system/house-worker.service`:
|
||
|
||
```ini
|
||
[Unit]
|
||
Description=House tracker worker
|
||
After=network-online.target
|
||
Wants=network-online.target
|
||
|
||
[Service]
|
||
User=stan
|
||
WorkingDirectory=/home/stan/house-worker
|
||
ExecStart=/home/stan/.bun/bin/bun --env-file=house.env scripts/house-worker.ts
|
||
Restart=always
|
||
RestartSec=60
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
```
|
||
|
||
```sh
|
||
sudo systemctl enable --now house-worker
|
||
journalctl -u house-worker -f
|
||
```
|
||
|
||
Or with Docker: `docker run -d --restart unless-stopped --name house-worker
|
||
--env-file house.env -v "$PWD":/app -w /app oven/bun:1.4.0 bun
|
||
scripts/house-worker.ts`.
|
||
|
||
## Worker settings
|
||
|
||
| variable | default |
|
||
| ------------------------ | ------------------------------------------------------------------------ |
|
||
| `HOUSE_FAST_SEARCHES` | `https://www.pararius.nl/huurwoningen/gouda/1300-1700` (comma separated) |
|
||
| `HOUSE_SLOW_SEARCHES` | rent + buy for the six cities above |
|
||
| `HOUSE_SLOW_EVERY_HOURS` | `12` |
|
||
| `HOUSE_SLOW_MAX_PAGES` | `80` per search |
|
||
| `HOUSE_MIN_GAP_SECONDS` | `8` between any two requests |
|
||
| `HOUSE_DAILY_BUDGET` | `1500` requests |
|
||
| `HOUSE_FETCH_MODE` | `fetch`, or `browser` (real Chromium) |
|
||
|
||
Any Pararius search URL works as a fast search: set the filters on the site
|
||
(price, radius, bedrooms) and paste the URL. Every extra fast search is one more
|
||
request per poll.
|
||
|
||
## When it gets challenged
|
||
|
||
You get one "worker was challenged" alert per 6 hours and the page header turns
|
||
red. In order:
|
||
|
||
1. Do nothing for a day. Backoff handles short blocks.
|
||
2. Slow down: raise `HOUSE_MIN_GAP_SECONDS`, lower `HOUSE_DAILY_BUDGET`, or drop
|
||
slow searches.
|
||
3. Save a page manually in your normal browser to investigate parsing offline.
|
||
Do not switch clients, use proxies or solve challenges to bypass a block.
|
||
|
||
The optional `HOUSE_FETCH_MODE=browser` uses headless Chromium with a persistent
|
||
profile (`npx playwright install chromium` once on the worker machine). It obeys
|
||
the same stop/backoff rules and is not a workaround for a challenged fetch.
|
||
|
||
## Parsers
|
||
|
||
`src/lib/house-tracker/pararius.ts`, tested against trimmed real pages in
|
||
`src/lib/house-tracker/fixtures/`. Listing pages expose an exact publish
|
||
timestamp (`odp.offered_since`), coordinates, deposit, energy label and agent.
|
||
`pararius-search-gouda.html` is a trimmed real search page saved on 2026-10-11,
|
||
with tracking and account data removed. It covers three listings, nested base
|
||
and total rent, action attributes and a lazy photo template. The synthetic
|
||
fixture still covers pagination and price on request. Rental `priceEur` is the
|
||
base/listed rent used by the search filter; a separately shown total can be
|
||
higher (Raam: €1,475 base, €1,750 total). If Pararius changes markup, the worker
|
||
reports `parse-error` in the worker log on the page.
|
||
|
||
## Adding listings by hand
|
||
|
||
`scripts/house-listing.ts` writes `house.listings` directly (links from chats,
|
||
screenshots). No restart or deploy is needed for new rows.
|
||
|
||
```sh
|
||
# DATABASE_URL or DB_URL must be set; locally: bun --env-file=.env run house …
|
||
bun run house add --url https://vbtverhuurmakelaars.nl/woning/gouda-vest-7-d \
|
||
--city Gouda --price "€ 1.422,- p.m." --area 84 --rooms 3 --bedrooms 2 --energy-label A
|
||
echo '{"url":"…","city":"Gouda","price_eur":1350,"area_m2":81}' | bun run house add --json -
|
||
bun run house add --street "Vest 7-D" --city Gouda --price 1422 # no link (screenshot)
|
||
bun run house update <id|url> --status viewing --notes "za 14:00" # "" clears a field
|
||
bun run house list [--all] [--city gouda] [--offer buy] [--json]
|
||
bun run house remove <id|url> [--hard] # soft remove marks it gone; restore undoes it
|
||
```
|
||
|
||
- `source` is the site (`funda`, `pararius`, `vbt`, `vesteda`, … or `manual`
|
||
without a link); `source_id` comes from the URL (Funda's number, Pararius'
|
||
worker key, otherwise the path), or a hash of the address. Adding the same
|
||
link again updates that row and clears `removed_at`.
|
||
- Fields follow `listingDetailSchema` (camelCase, snake_case or the flag names)
|
||
and are lenient: "€ 1.422,- p.m." and "84 m²" are parsed. Only given fields are
|
||
written. `city_slug` is derived from `city` in Pararius' slug style.
|
||
- `--first-seen` sets when it was shared; price changes go to `price_history`.
|
||
- Manual rows use `first_seen_lane = 'manual'` and never trigger alerts.
|