stan/personal-website
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
house-tracker
personal-website/docs/house-tracker.md
Stan Runge b52e9188f4 house worker
2026-10-11 22:05:05 +00:00

8.2 KiB
Raw Permalink Blame History

House tracker

Personal tool: collects Dutch housing listings into Postgres, shows them on /projects/house-tracker (admin only) and sends an alert when a new listing matches a rule. Only Pararius for now; other sources plug in the same way.

How it fits together

home box (residential IP)                     stanrunge.dev (Hetzner)
┌──────────────────────────┐   POST sync     ┌──────────────────────────────┐
│ scripts/house-worker.ts  │ ──────────────▶ │ /projects/house-tracker/api/ │
│  polls pararius.nl       │   POST details  │   worker/{sync,details,report}│
│  parses HTML             │ ◀────────────── │  stores, dedups, alerts      │
└──────────────────────────┘  needDetails    └──────────────┬───────────────┘
                                                            ▼
                                              Discord webhook / ntfy push

The server never talks to the listing sites. Pararius sits behind Cloudflare, which challenges datacenter IPs and plain HTTP clients, so the worker runs at home where traffic looks like a normal visitor.

Two lanes:

  • fast: one filtered search (default huurwoningen/gouda/1300-1700), page 1 only. Every 3–5 min on weekdays 07:00–18:00 Amsterdam time, every 30–60 min otherwise. Listings near Gouda were all published in office hours on weekdays. Alerts are sent from the search card alone, before the detail page is fetched.
  • slow: full, paginated sweeps of rent and buy listings in Gouda, Rotterdam, Den Haag, Utrecht, Amsterdam and Eindhoven every 12 h. Unfiltered city sweeps also mark listings that disappeared, which gives "days listed" in the stats.

Safety rules in the worker: one request at a time, at least 8 s apart with jitter, a daily budget (1500 by default, the slow lane may only use 60%), and any challenge/403/429 stops everything and backs off 15 min → 30 → 60 … 6 h. The first poll of any search only seeds the database, so you don't get 40 alerts for listings that were already online.

Server setup

Add to the production env file (/etc/personal-website/prod.env):

# openssl rand -hex 32
HOUSE_WORKER_TOKEN=
# one or both
HOUSE_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
HOUSE_NTFY_URL=https://ntfy.sh/some-long-random-topic
HOUSE_NTFY_TOKEN=            # only for a protected ntfy topic
PUBLIC_MAPTILER_API_KEY=     # already set for Animaldex

scripts/migrate-house-tracker.ts runs on every deploy (idempotent). Use "send test alert" on the page to check the alert channel.

If Cloudflare Bot Fight Mode / WAF challenges are on for stanrunge.dev, add a WAF skip rule for URI Path starts with /projects/house-tracker/api/worker/, otherwise the worker's POSTs get challenged too. The endpoints are protected by the bearer token.

Development behind Cloudflare Access

For https://dev.stanrunge.dev, keep Access enabled. In Cloudflare Zero Trust, go to Access controls → Service credentials → Service Tokens and create a token named house-worker. Save its Client ID and Client Secret. In the Access application protecting the dev hostname, add a Service Auth policy with Include → Service Token → house-worker. Keep the existing browser login policy. See Cloudflare's service token documentation.

Set HOUSE_WORKER_TOKEN in the dev website's .env and restart its dev server. On the home machine, copy house.env.example to house.env, fill in the same worker token plus the Cloudflare Client ID and Client Secret, then run:

chmod 600 house.env
bun --env-file=house.env scripts/house-worker.ts --check-api

This makes one authenticated GET to the website's worker health endpoint. It does not contact Pararius, sync listings or send alerts. The expected output is API connection OK. Access credentials are sent only to the website API; the client refuses redirects and stops immediately on an authentication failure. An HTTP 503 can mean the website's worker token is missing; an HTTP 404 can mean the running website does not yet include the health route.

Home worker setup

Needs a machine that stays on (old laptop, Raspberry Pi, NAS) with Bun 1.4 or newer; older versions cannot read this repository's lockfile. Use bun upgrade if needed.

git clone https://git.stanrunge.dev/stan/personal-website ~/house-worker
cd ~/house-worker && bun install --frozen-lockfile
cat > house.env <<'EOF'
HOUSE_API_URL=https://stanrunge.dev
HOUSE_WORKER_TOKEN=same-value-as-the-server
EOF
chmod 600 house.env

# 1. check parsing without sending anything
bun --env-file=house.env scripts/house-worker.ts --dry-run --once
# 2. one real poll
bun --env-file=house.env scripts/house-worker.ts --once

Then run it as a service, /etc/systemd/system/house-worker.service:

[Unit]
Description=House tracker worker
After=network-online.target
Wants=network-online.target

[Service]
User=stan
WorkingDirectory=/home/stan/house-worker
ExecStart=/home/stan/.bun/bin/bun --env-file=house.env scripts/house-worker.ts
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now house-worker
journalctl -u house-worker -f

Or with Docker: docker run -d --restart unless-stopped --name house-worker --env-file house.env -v "$PWD":/app -w /app oven/bun:1.4.0 bun scripts/house-worker.ts.

Worker settings

variable default
HOUSE_FAST_SEARCHES https://www.pararius.nl/huurwoningen/gouda/1300-1700 (comma separated)
HOUSE_SLOW_SEARCHES rent + buy for the six cities above
HOUSE_SLOW_EVERY_HOURS 12
HOUSE_SLOW_MAX_PAGES 80 per search
HOUSE_MIN_GAP_SECONDS 8 between any two requests
HOUSE_DAILY_BUDGET 1500 requests
HOUSE_FETCH_MODE fetch, or browser (real Chromium)

Any Pararius search URL works as a fast search: set the filters on the site (price, radius, bedrooms) and paste the URL. Every extra fast search is one more request per poll.

When it gets challenged

You get one "worker was challenged" alert per 6 hours and the page header turns red. In order:

  1. Do nothing for a day. Backoff handles short blocks.
  2. Slow down: raise HOUSE_MIN_GAP_SECONDS, lower HOUSE_DAILY_BUDGET, or drop slow searches.
  3. Save a page manually in your normal browser to investigate parsing offline. Do not switch clients, use proxies or solve challenges to bypass a block.

The optional HOUSE_FETCH_MODE=browser uses headless Chromium with a persistent profile (npx playwright install chromium once on the worker machine). It obeys the same stop/backoff rules and is not a workaround for a challenged fetch.

Parsers

src/lib/house-tracker/pararius.ts, tested against trimmed real pages in src/lib/house-tracker/fixtures/. Listing pages expose an exact publish timestamp (odp.offered_since), coordinates, deposit, energy label and agent. pararius-search-gouda.html is a trimmed real search page saved on 2026-10-11, with tracking and account data removed. It covers three listings, nested base and total rent, action attributes and a lazy photo template. The synthetic fixture still covers pagination and price on request. Rental priceEur is the base/listed rent used by the search filter; a separately shown total can be higher (Raam: €1,475 base, €1,750 total). If Pararius changes markup, the worker reports parse-error in the worker log on the page.