# House tracker

Personal tool: collects Dutch housing listings into Postgres, shows them on
`/projects/house-tracker` (admin only) and sends an alert when a new listing
matches a rule. Only Pararius for now; other sources plug in the same way.

## How it fits together

```
home box (residential IP)                     stanrunge.dev (Hetzner)
┌──────────────────────────┐   POST sync     ┌──────────────────────────────┐
│ scripts/house-worker.ts  │ ──────────────▶ │ /projects/house-tracker/api/ │
│  polls pararius.nl       │   POST details  │   worker/{sync,details,report}│
│  parses HTML             │ ◀────────────── │  stores, dedups, alerts      │
└──────────────────────────┘  needDetails    └──────────────┬───────────────┘
                                                            ▼
                                              Discord webhook / ntfy push
```

The server never talks to the listing sites. Pararius sits behind Cloudflare,
which challenges datacenter IPs and plain HTTP clients, so the worker runs at
home where traffic looks like a normal visitor.

Two lanes:

- **fast**: one filtered search (default `huurwoningen/gouda/1300-1700`), page 1
  only. Every 3–5 min on weekdays 07:00–18:00 Amsterdam time, every 30–60 min
  otherwise. Listings near Gouda were all published in office hours on weekdays.
  Alerts are sent from the search card alone, before the detail page is fetched.
- **slow**: full, paginated sweeps of rent and buy listings in Gouda, Rotterdam,
  Den Haag, Utrecht, Amsterdam and Eindhoven every 12 h. Unfiltered city sweeps
  also mark listings that disappeared, which gives "days listed" in the stats.

Safety rules in the worker: one request at a time, at least 8 s apart with
jitter, a daily budget (1500 by default, the slow lane may only use 60%), and
any challenge/403/429 stops everything and backs off 15 min → 30 → 60 … 6 h.
The first poll of any search only seeds the database, so you don't get 40
alerts for listings that were already online.

## Server setup

Add to the production env file (`/etc/personal-website/prod.env`):

```text
# openssl rand -hex 32
HOUSE_WORKER_TOKEN=
# one or both
HOUSE_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
HOUSE_NTFY_URL=https://ntfy.sh/some-long-random-topic
HOUSE_NTFY_TOKEN=            # only for a protected ntfy topic
PUBLIC_MAPTILER_API_KEY=     # already set for Animaldex
```

`scripts/migrate-house-tracker.ts` runs on every deploy (idempotent). Use
"send test alert" on the page to check the alert channel.

If Cloudflare Bot Fight Mode / WAF challenges are on for stanrunge.dev, add a
WAF skip rule for `URI Path starts with /projects/house-tracker/api/worker/`,
otherwise the worker's POSTs get challenged too. The endpoints are protected by
the bearer token.

## Home worker setup

Needs a machine that stays on (old laptop, Raspberry Pi, NAS) with bun.

```sh
git clone https://git.stanrunge.dev/stan/personal-website ~/house-worker
cd ~/house-worker && bun install
cat > house.env <<'EOF'
HOUSE_API_URL=https://stanrunge.dev
HOUSE_WORKER_TOKEN=same-value-as-the-server
EOF
chmod 600 house.env

# 1. check parsing without sending anything
bun --env-file=house.env scripts/house-worker.ts --dry-run --once
# 2. one real poll
bun --env-file=house.env scripts/house-worker.ts --once
```

Then run it as a service, `/etc/systemd/system/house-worker.service`:

```ini
[Unit]
Description=House tracker worker
After=network-online.target
Wants=network-online.target

[Service]
User=stan
WorkingDirectory=/home/stan/house-worker
ExecStart=/home/stan/.bun/bin/bun --env-file=house.env scripts/house-worker.ts
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target
```

```sh
sudo systemctl enable --now house-worker
journalctl -u house-worker -f
```

Or with Docker: `docker run -d --restart unless-stopped --name house-worker
--env-file house.env -v "$PWD":/app -w /app oven/bun:1.4.0 bun
scripts/house-worker.ts`.

## Worker settings

| variable                 | default                                                                  |
| ------------------------ | ------------------------------------------------------------------------ |
| `HOUSE_FAST_SEARCHES`    | `https://www.pararius.nl/huurwoningen/gouda/1300-1700` (comma separated) |
| `HOUSE_SLOW_SEARCHES`    | rent + buy for the six cities above                                      |
| `HOUSE_SLOW_EVERY_HOURS` | `12`                                                                     |
| `HOUSE_SLOW_MAX_PAGES`   | `80` per search                                                          |
| `HOUSE_MIN_GAP_SECONDS`  | `8` between any two requests                                             |
| `HOUSE_DAILY_BUDGET`     | `1500` requests                                                          |
| `HOUSE_FETCH_MODE`       | `fetch`, or `browser` (real Chromium)                                    |

Any Pararius search URL works as a fast search: set the filters on the site
(price, radius, bedrooms) and paste the URL. Every extra fast search is one more
request per poll.

## When it gets challenged

You get one "worker was challenged" alert per 6 hours and the page header turns
red. In order:

1. Do nothing for a day. Backoff handles short blocks.
2. Slow down: raise `HOUSE_MIN_GAP_SECONDS`, lower `HOUSE_DAILY_BUDGET`, or drop
   slow searches.
3. `HOUSE_FETCH_MODE=browser` uses headless Chromium with a persistent profile
   (`npx playwright install chromium` once on the worker machine). Heavier, but
   passes checks that plain HTTP clients don't.

Commercial VPN exits are usually datacenter IPs and get challenged more, not less.

## Parsers

`src/lib/house-tracker/pararius.ts`, tested against trimmed real pages in
`src/lib/house-tracker/fixtures/`. Listing pages expose an exact publish
timestamp (`odp.offered_since`), coordinates, deposit, energy label and agent.
The search-page fixture is hand-written; replace it with a real saved page when
the markup is confirmed. If Pararius changes markup, the worker reports
`parse-error` in the worker log on the page.