179 lines
8.2 KiB
Markdown
179 lines
8.2 KiB
Markdown
# House tracker
|
||
|
||
Personal tool: collects Dutch housing listings into Postgres, shows them on
|
||
`/projects/house-tracker` (admin only) and sends an alert when a new listing
|
||
matches a rule. Only Pararius for now; other sources plug in the same way.
|
||
|
||
## How it fits together
|
||
|
||
```
|
||
home box (residential IP) stanrunge.dev (Hetzner)
|
||
┌──────────────────────────┐ POST sync ┌──────────────────────────────┐
|
||
│ scripts/house-worker.ts │ ──────────────▶ │ /projects/house-tracker/api/ │
|
||
│ polls pararius.nl │ POST details │ worker/{sync,details,report}│
|
||
│ parses HTML │ ◀────────────── │ stores, dedups, alerts │
|
||
└──────────────────────────┘ needDetails └──────────────┬───────────────┘
|
||
▼
|
||
Discord webhook / ntfy push
|
||
```
|
||
|
||
The server never talks to the listing sites. Pararius sits behind Cloudflare,
|
||
which challenges datacenter IPs and plain HTTP clients, so the worker runs at
|
||
home where traffic looks like a normal visitor.
|
||
|
||
Two lanes:
|
||
|
||
- **fast**: one filtered search (default `huurwoningen/gouda/1300-1700`), page 1
|
||
only. Every 3–5 min on weekdays 07:00–18:00 Amsterdam time, every 30–60 min
|
||
otherwise. Listings near Gouda were all published in office hours on weekdays.
|
||
Alerts are sent from the search card alone, before the detail page is fetched.
|
||
- **slow**: full, paginated sweeps of rent and buy listings in Gouda, Rotterdam,
|
||
Den Haag, Utrecht, Amsterdam and Eindhoven every 12 h. Unfiltered city sweeps
|
||
also mark listings that disappeared, which gives "days listed" in the stats.
|
||
|
||
Safety rules in the worker: one request at a time, at least 8 s apart with
|
||
jitter, a daily budget (1500 by default, the slow lane may only use 60%), and
|
||
any challenge/403/429 stops everything and backs off 15 min → 30 → 60 … 6 h.
|
||
The first poll of any search only seeds the database, so you don't get 40
|
||
alerts for listings that were already online.
|
||
|
||
## Server setup
|
||
|
||
Add to the production env file (`/etc/personal-website/prod.env`):
|
||
|
||
```text
|
||
# openssl rand -hex 32
|
||
HOUSE_WORKER_TOKEN=
|
||
# one or both
|
||
HOUSE_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
|
||
HOUSE_NTFY_URL=https://ntfy.sh/some-long-random-topic
|
||
HOUSE_NTFY_TOKEN= # only for a protected ntfy topic
|
||
PUBLIC_MAPTILER_API_KEY= # already set for Animaldex
|
||
```
|
||
|
||
`scripts/migrate-house-tracker.ts` runs on every deploy (idempotent). Use
|
||
"send test alert" on the page to check the alert channel.
|
||
|
||
If Cloudflare Bot Fight Mode / WAF challenges are on for stanrunge.dev, add a
|
||
WAF skip rule for `URI Path starts with /projects/house-tracker/api/worker/`,
|
||
otherwise the worker's POSTs get challenged too. The endpoints are protected by
|
||
the bearer token.
|
||
|
||
### Development behind Cloudflare Access
|
||
|
||
For `https://dev.stanrunge.dev`, keep Access enabled. In Cloudflare Zero Trust,
|
||
go to **Access controls → Service credentials → Service Tokens** and create a
|
||
token named `house-worker`. Save its Client ID and Client Secret. In the Access
|
||
application protecting the dev hostname, add a **Service Auth** policy with
|
||
**Include → Service Token → house-worker**. Keep the existing browser login
|
||
policy. See [Cloudflare's service token documentation](https://developers.cloudflare.com/cloudflare-one/access-controls/service-credentials/service-tokens/).
|
||
|
||
Set `HOUSE_WORKER_TOKEN` in the dev website's `.env` and restart its dev server.
|
||
On the home machine, copy `house.env.example` to `house.env`, fill in the same
|
||
worker token plus the Cloudflare Client ID and Client Secret, then run:
|
||
|
||
```sh
|
||
chmod 600 house.env
|
||
bun --env-file=house.env scripts/house-worker.ts --check-api
|
||
```
|
||
|
||
This makes one authenticated GET to the website's worker health endpoint. It
|
||
does not contact Pararius, sync listings or send alerts. The expected output is
|
||
`API connection OK`. Access credentials are sent only to the website API; the
|
||
client refuses redirects and stops immediately on an authentication failure.
|
||
An HTTP 503 can mean the website's worker token is missing; an HTTP 404 can mean
|
||
the running website does not yet include the health route.
|
||
|
||
## Home worker setup
|
||
|
||
Needs a machine that stays on (old laptop, Raspberry Pi, NAS) with Bun 1.4 or
|
||
newer; older versions cannot read this repository's lockfile. Use `bun upgrade`
|
||
if needed.
|
||
|
||
```sh
|
||
git clone https://git.stanrunge.dev/stan/personal-website ~/house-worker
|
||
cd ~/house-worker && bun install --frozen-lockfile
|
||
cat > house.env <<'EOF'
|
||
HOUSE_API_URL=https://stanrunge.dev
|
||
HOUSE_WORKER_TOKEN=same-value-as-the-server
|
||
EOF
|
||
chmod 600 house.env
|
||
|
||
# 1. check parsing without sending anything
|
||
bun --env-file=house.env scripts/house-worker.ts --dry-run --once
|
||
# 2. one real poll
|
||
bun --env-file=house.env scripts/house-worker.ts --once
|
||
```
|
||
|
||
Then run it as a service, `/etc/systemd/system/house-worker.service`:
|
||
|
||
```ini
|
||
[Unit]
|
||
Description=House tracker worker
|
||
After=network-online.target
|
||
Wants=network-online.target
|
||
|
||
[Service]
|
||
User=stan
|
||
WorkingDirectory=/home/stan/house-worker
|
||
ExecStart=/home/stan/.bun/bin/bun --env-file=house.env scripts/house-worker.ts
|
||
Restart=always
|
||
RestartSec=60
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
```
|
||
|
||
```sh
|
||
sudo systemctl enable --now house-worker
|
||
journalctl -u house-worker -f
|
||
```
|
||
|
||
Or with Docker: `docker run -d --restart unless-stopped --name house-worker
|
||
--env-file house.env -v "$PWD":/app -w /app oven/bun:1.4.0 bun
|
||
scripts/house-worker.ts`.
|
||
|
||
## Worker settings
|
||
|
||
| variable | default |
|
||
| ------------------------ | ------------------------------------------------------------------------ |
|
||
| `HOUSE_FAST_SEARCHES` | `https://www.pararius.nl/huurwoningen/gouda/1300-1700` (comma separated) |
|
||
| `HOUSE_SLOW_SEARCHES` | rent + buy for the six cities above |
|
||
| `HOUSE_SLOW_EVERY_HOURS` | `12` |
|
||
| `HOUSE_SLOW_MAX_PAGES` | `80` per search |
|
||
| `HOUSE_MIN_GAP_SECONDS` | `8` between any two requests |
|
||
| `HOUSE_DAILY_BUDGET` | `1500` requests |
|
||
| `HOUSE_FETCH_MODE` | `fetch`, or `browser` (real Chromium) |
|
||
|
||
Any Pararius search URL works as a fast search: set the filters on the site
|
||
(price, radius, bedrooms) and paste the URL. Every extra fast search is one more
|
||
request per poll.
|
||
|
||
## When it gets challenged
|
||
|
||
You get one "worker was challenged" alert per 6 hours and the page header turns
|
||
red. In order:
|
||
|
||
1. Do nothing for a day. Backoff handles short blocks.
|
||
2. Slow down: raise `HOUSE_MIN_GAP_SECONDS`, lower `HOUSE_DAILY_BUDGET`, or drop
|
||
slow searches.
|
||
3. Save a page manually in your normal browser to investigate parsing offline.
|
||
Do not switch clients, use proxies or solve challenges to bypass a block.
|
||
|
||
The optional `HOUSE_FETCH_MODE=browser` uses headless Chromium with a persistent
|
||
profile (`npx playwright install chromium` once on the worker machine). It obeys
|
||
the same stop/backoff rules and is not a workaround for a challenged fetch.
|
||
|
||
## Parsers
|
||
|
||
`src/lib/house-tracker/pararius.ts`, tested against trimmed real pages in
|
||
`src/lib/house-tracker/fixtures/`. Listing pages expose an exact publish
|
||
timestamp (`odp.offered_since`), coordinates, deposit, energy label and agent.
|
||
`pararius-search-gouda.html` is a trimmed real search page saved on 2026-10-11,
|
||
with tracking and account data removed. It covers three listings, nested base
|
||
and total rent, action attributes and a lazy photo template. The synthetic
|
||
fixture still covers pagination and price on request. Rental `priceEur` is the
|
||
base/listed rent used by the search filter; a separately shown total can be
|
||
higher (Raam: €1,475 base, €1,750 total). If Pararius changes markup, the worker
|
||
reports `parse-error` in the worker log on the page.
|