# House tracker

Personal tool: collects Dutch housing listings into Postgres, shows them on
`/projects/house-tracker` (admin only) and sends an alert when a new listing
matches a rule. Only Pararius for now; other sources plug in the same way.

## How it fits together

```
home box (residential IP)                     stanrunge.dev (Hetzner)
┌──────────────────────────┐   POST sync     ┌──────────────────────────────┐
│ scripts/house-worker.ts  │ ──────────────▶ │ /projects/house-tracker/api/ │
│  polls pararius.nl       │   POST details  │   worker/{sync,details,report}│
│  parses HTML             │ ◀────────────── │  stores, dedups, alerts      │
└──────────────────────────┘  needDetails    └──────────────┬───────────────┘
                                                            ▼
                                              Discord webhook / ntfy push
```

The server never talks to the listing sites. Pararius sits behind Cloudflare,
which challenges datacenter IPs and plain HTTP clients, so the worker runs at
home where traffic looks like a normal visitor.

Two lanes:

- **fast**: one filtered search (default `huurwoningen/gouda/1300-1700`), page 1
  only. Every 3–5 min on weekdays 07:00–18:00 Amsterdam time, every 30–60 min
  otherwise. Listings near Gouda were all published in office hours on weekdays.
  Alerts are sent from the search card alone, before the detail page is fetched.
- **slow**: full, paginated sweeps of rent and buy listings in Gouda, Rotterdam,
  Den Haag, Utrecht, Amsterdam and Eindhoven every 12 h. Unfiltered city sweeps
  also mark listings that disappeared, which gives "days listed" in the stats.

Safety rules in the worker: one request at a time, at least 8 s apart with
jitter, a daily budget (1500 by default, the slow lane may only use 60%), and
any challenge/403/429 stops everything and backs off 15 min → 30 → 60 … 6 h.
The first poll of any search only seeds the database, so you don't get 40
alerts for listings that were already online.

## Server setup

Add to the production env file (`/etc/personal-website/prod.env`):

```text
# openssl rand -hex 32
HOUSE_WORKER_TOKEN=
# one or both
HOUSE_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
HOUSE_NTFY_URL=https://ntfy.sh/some-long-random-topic
HOUSE_NTFY_TOKEN=            # only for a protected ntfy topic
PUBLIC_MAPTILER_API_KEY=     # already set for Animaldex
```

`scripts/migrate-house-tracker.ts` runs on every deploy (idempotent). Use
"send test alert" on the page to check the alert channel.

If Cloudflare Bot Fight Mode / WAF challenges are on for stanrunge.dev, add a
WAF skip rule for `URI Path starts with /projects/house-tracker/api/worker/`,
otherwise the worker's POSTs get challenged too. The endpoints are protected by
the bearer token.

### Development behind Cloudflare Access

For `https://dev.stanrunge.dev`, keep Access enabled. In Cloudflare Zero Trust,
go to **Access controls → Service credentials → Service Tokens** and create a
token named `house-worker`. Save its Client ID and Client Secret. In the Access
application protecting the dev hostname, add a **Service Auth** policy with
**Include → Service Token → house-worker**. Keep the existing browser login
policy. See [Cloudflare's service token documentation](https://developers.cloudflare.com/cloudflare-one/access-controls/service-credentials/service-tokens/).

Set `HOUSE_WORKER_TOKEN` in the dev website's `.env` and restart its dev server.
On the home machine, copy `house.env.example` to `house.env`, fill in the same
worker token plus the Cloudflare Client ID and Client Secret, then run:

```sh
chmod 600 house.env
bun --env-file=house.env scripts/house-worker.ts --check-api
```

This makes one authenticated GET to the website's worker health endpoint. It
does not contact Pararius, sync listings or send alerts. The expected output is
`API connection OK`. Access credentials are sent only to the website API; the
client refuses redirects and stops immediately on an authentication failure.
An HTTP 503 can mean the website's worker token is missing; an HTTP 404 can mean
the running website does not yet include the health route.

## Home worker setup

Needs a machine that stays on (old laptop, Raspberry Pi, NAS) with Bun 1.4 or
newer; older versions cannot read this repository's lockfile. Use `bun upgrade`
if needed.

```sh
git clone https://git.stanrunge.dev/stan/personal-website ~/house-worker
cd ~/house-worker && bun install --frozen-lockfile
cat > house.env <<'EOF'
HOUSE_API_URL=https://stanrunge.dev
HOUSE_WORKER_TOKEN=same-value-as-the-server
EOF
chmod 600 house.env

# 1. check parsing without sending anything
bun --env-file=house.env scripts/house-worker.ts --dry-run --once
# 2. one real poll
bun --env-file=house.env scripts/house-worker.ts --once
```

Then run it as a service, `/etc/systemd/system/house-worker.service`:

```ini
[Unit]
Description=House tracker worker
After=network-online.target
Wants=network-online.target

[Service]
User=stan
WorkingDirectory=/home/stan/house-worker
ExecStart=/home/stan/.bun/bin/bun --env-file=house.env scripts/house-worker.ts
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target
```

```sh
sudo systemctl enable --now house-worker
journalctl -u house-worker -f
```

Or with Docker: `docker run -d --restart unless-stopped --name house-worker
--env-file house.env -v "$PWD":/app -w /app oven/bun:1.4.0 bun
scripts/house-worker.ts`.

## Worker settings

| variable                 | default                                                                  |
| ------------------------ | ------------------------------------------------------------------------ |
| `HOUSE_FAST_SEARCHES`    | `https://www.pararius.nl/huurwoningen/gouda/1300-1700` (comma separated) |
| `HOUSE_SLOW_SEARCHES`    | rent + buy for the six cities above                                      |
| `HOUSE_SLOW_EVERY_HOURS` | `12`                                                                     |
| `HOUSE_SLOW_MAX_PAGES`   | `80` per search                                                          |
| `HOUSE_MIN_GAP_SECONDS`  | `8` between any two requests                                             |
| `HOUSE_DAILY_BUDGET`     | `1500` requests                                                          |
| `HOUSE_FETCH_MODE`       | `fetch`, or `browser` (real Chromium)                                    |

Any Pararius search URL works as a fast search: set the filters on the site
(price, radius, bedrooms) and paste the URL. Every extra fast search is one more
request per poll.

## When it gets challenged

You get one "worker was challenged" alert per 6 hours and the page header turns
red. In order:

1. Do nothing for a day. Backoff handles short blocks.
2. Slow down: raise `HOUSE_MIN_GAP_SECONDS`, lower `HOUSE_DAILY_BUDGET`, or drop
   slow searches.
3. Save a page manually in your normal browser to investigate parsing offline.
   Do not switch clients, use proxies or solve challenges to bypass a block.

The optional `HOUSE_FETCH_MODE=browser` uses headless Chromium with a persistent
profile (`npx playwright install chromium` once on the worker machine). It obeys
the same stop/backoff rules and is not a workaround for a challenged fetch.

## Parsers

`src/lib/house-tracker/pararius.ts`, tested against trimmed real pages in
`src/lib/house-tracker/fixtures/`. Listing pages expose an exact publish
timestamp (`odp.offered_since`), coordinates, deposit, energy label and agent.
`pararius-search-gouda.html` is a trimmed real search page saved on 2026-10-11,
with tracking and account data removed. It covers three listings, nested base
and total rent, action attributes and a lazy photo template. The synthetic
fixture still covers pagination and price on request. Rental `priceEur` is the
base/listed rent used by the search filter; a separately shown total can be
higher (Raam: €1,475 base, €1,750 total). If Pararius changes markup, the worker
reports `parse-error` in the worker log on the page.