feat: house tracker with home worker, alerts and listing map
Collects Pararius rent/buy listings into Postgres via a worker that runs on a residential connection, sends Discord/ntfy alerts for new listings matching rules, and shows an admin-only map, market stats and worker log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013HfF5tnChi8tkCx9svJP6D
This commit is contained in:
146
docs/house-tracker.md
Normal file
146
docs/house-tracker.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# House tracker
|
||||
|
||||
Personal tool: collects Dutch housing listings into Postgres, shows them on
|
||||
`/projects/house-tracker` (admin only) and sends an alert when a new listing
|
||||
matches a rule. Only Pararius for now; other sources plug in the same way.
|
||||
|
||||
## How it fits together
|
||||
|
||||
```
|
||||
home box (residential IP) stanrunge.dev (Hetzner)
|
||||
┌──────────────────────────┐ POST sync ┌──────────────────────────────┐
|
||||
│ scripts/house-worker.ts │ ──────────────▶ │ /projects/house-tracker/api/ │
|
||||
│ polls pararius.nl │ POST details │ worker/{sync,details,report}│
|
||||
│ parses HTML │ ◀────────────── │ stores, dedups, alerts │
|
||||
└──────────────────────────┘ needDetails └──────────────┬───────────────┘
|
||||
▼
|
||||
Discord webhook / ntfy push
|
||||
```
|
||||
|
||||
The server never talks to the listing sites. Pararius sits behind Cloudflare,
|
||||
which challenges datacenter IPs and plain HTTP clients, so the worker runs at
|
||||
home where traffic looks like a normal visitor.
|
||||
|
||||
Two lanes:
|
||||
|
||||
- **fast**: one filtered search (default `huurwoningen/gouda/1300-1700`), page 1
|
||||
only. Every 3–5 min on weekdays 07:00–18:00 Amsterdam time, every 30–60 min
|
||||
otherwise. Listings near Gouda were all published in office hours on weekdays.
|
||||
Alerts are sent from the search card alone, before the detail page is fetched.
|
||||
- **slow**: full, paginated sweeps of rent and buy listings in Gouda, Rotterdam,
|
||||
Den Haag, Utrecht, Amsterdam and Eindhoven every 12 h. Unfiltered city sweeps
|
||||
also mark listings that disappeared, which gives "days listed" in the stats.
|
||||
|
||||
Safety rules in the worker: one request at a time, at least 8 s apart with
|
||||
jitter, a daily budget (1500 by default, the slow lane may only use 60%), and
|
||||
any challenge/403/429 stops everything and backs off 15 min → 30 → 60 … 6 h.
|
||||
The first poll of any search only seeds the database, so you don't get 40
|
||||
alerts for listings that were already online.
|
||||
|
||||
## Server setup
|
||||
|
||||
Add to the production env file (`/etc/personal-website/prod.env`):
|
||||
|
||||
```text
|
||||
# openssl rand -hex 32
|
||||
HOUSE_WORKER_TOKEN=
|
||||
# one or both
|
||||
HOUSE_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
|
||||
HOUSE_NTFY_URL=https://ntfy.sh/some-long-random-topic
|
||||
HOUSE_NTFY_TOKEN= # only for a protected ntfy topic
|
||||
PUBLIC_MAPTILER_API_KEY= # already set for Animaldex
|
||||
```
|
||||
|
||||
`scripts/migrate-house-tracker.ts` runs on every deploy (idempotent). Use
|
||||
"send test alert" on the page to check the alert channel.
|
||||
|
||||
If Cloudflare Bot Fight Mode / WAF challenges are on for stanrunge.dev, add a
|
||||
WAF skip rule for `URI Path starts with /projects/house-tracker/api/worker/`,
|
||||
otherwise the worker's POSTs get challenged too. The endpoints are protected by
|
||||
the bearer token.
|
||||
|
||||
## Home worker setup
|
||||
|
||||
Needs a machine that stays on (old laptop, Raspberry Pi, NAS) with bun.
|
||||
|
||||
```sh
|
||||
git clone https://git.stanrunge.dev/stan/personal-website ~/house-worker
|
||||
cd ~/house-worker && bun install
|
||||
cat > house.env <<'EOF'
|
||||
HOUSE_API_URL=https://stanrunge.dev
|
||||
HOUSE_WORKER_TOKEN=same-value-as-the-server
|
||||
EOF
|
||||
chmod 600 house.env
|
||||
|
||||
# 1. check parsing without sending anything
|
||||
bun --env-file=house.env scripts/house-worker.ts --dry-run --once
|
||||
# 2. one real poll
|
||||
bun --env-file=house.env scripts/house-worker.ts --once
|
||||
```
|
||||
|
||||
Then run it as a service, `/etc/systemd/system/house-worker.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=House tracker worker
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=stan
|
||||
WorkingDirectory=/home/stan/house-worker
|
||||
ExecStart=/home/stan/.bun/bin/bun --env-file=house.env scripts/house-worker.ts
|
||||
Restart=always
|
||||
RestartSec=60
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now house-worker
|
||||
journalctl -u house-worker -f
|
||||
```
|
||||
|
||||
Or with Docker: `docker run -d --restart unless-stopped --name house-worker
|
||||
--env-file house.env -v "$PWD":/app -w /app oven/bun:1.4.0 bun
|
||||
scripts/house-worker.ts`.
|
||||
|
||||
## Worker settings
|
||||
|
||||
| variable | default |
|
||||
| ------------------------ | ------------------------------------------------------------------------ |
|
||||
| `HOUSE_FAST_SEARCHES` | `https://www.pararius.nl/huurwoningen/gouda/1300-1700` (comma separated) |
|
||||
| `HOUSE_SLOW_SEARCHES` | rent + buy for the six cities above |
|
||||
| `HOUSE_SLOW_EVERY_HOURS` | `12` |
|
||||
| `HOUSE_SLOW_MAX_PAGES` | `80` per search |
|
||||
| `HOUSE_MIN_GAP_SECONDS` | `8` between any two requests |
|
||||
| `HOUSE_DAILY_BUDGET` | `1500` requests |
|
||||
| `HOUSE_FETCH_MODE` | `fetch`, or `browser` (real Chromium) |
|
||||
|
||||
Any Pararius search URL works as a fast search: set the filters on the site
|
||||
(price, radius, bedrooms) and paste the URL. Every extra fast search is one more
|
||||
request per poll.
|
||||
|
||||
## When it gets challenged
|
||||
|
||||
You get one "worker was challenged" alert per 6 hours and the page header turns
|
||||
red. In order:
|
||||
|
||||
1. Do nothing for a day. Backoff handles short blocks.
|
||||
2. Slow down: raise `HOUSE_MIN_GAP_SECONDS`, lower `HOUSE_DAILY_BUDGET`, or drop
|
||||
slow searches.
|
||||
3. `HOUSE_FETCH_MODE=browser` uses headless Chromium with a persistent profile
|
||||
(`npx playwright install chromium` once on the worker machine). Heavier, but
|
||||
passes checks that plain HTTP clients don't.
|
||||
|
||||
Commercial VPN exits are usually datacenter IPs and get challenged more, not less.
|
||||
|
||||
## Parsers
|
||||
|
||||
`src/lib/house-tracker/pararius.ts`, tested against trimmed real pages in
|
||||
`src/lib/house-tracker/fixtures/`. Listing pages expose an exact publish
|
||||
timestamp (`odp.offered_since`), coordinates, deposit, energy label and agent.
|
||||
The search-page fixture is hand-written; replace it with a real saved page when
|
||||
the markup is confirmed. If Pararius changes markup, the worker reports
|
||||
`parse-error` in the worker log on the page.
|
||||
Reference in New Issue
Block a user