Expand website and game trackers

README.md

@@ -7,6 +7,86 @@ This is my personal website, including:
 - Blog posts + comments
 - Projects like Animaldex, CatchEmAll
 
+## osu! Map Finder
+
+Available at `/projects/spotify-beatmap-finder`. Built with the existing Svelte 5,
+SvelteKit, Tailwind 4, and adapter-node setup. Uses server-only direct `fetch`
+calls; no Spotify/osu! SDK or additional services.
+
+Set these private variables in your local `.env` or deployment environment:
+
+```text
+SPOTIFY_CLIENT_ID
+SPOTIFY_CLIENT_SECRET
+OSU_CLIENT_ID
+OSU_CLIENT_SECRET
+```
+
+Never prefix these with `PUBLIC_` or commit `.env`. Providers read
+`$env/dynamic/private`; the browser receives only display metadata and results.
+The other projects on this personal site retain their own environment requirements.
+
+Supported inputs: song text (`Deftones Change`), Spotify track, playlist, and album
+URLs, and `spotify:track:ID`, `spotify:playlist:ID`, `spotify:album:ID` URIs.
+URLs allow HTTP/HTTPS, `www`, locale prefixes, query parameters and trailing slashes.
+Artist, episode, show and other resources receive a validation error.
+
+Playlists and albums preserve order and process up to **50 usable unique tracks**
+(`MAX_PLAYLIST_TRACKS`), scanning at most **10 pages** (`MAX_PLAYLIST_PAGES`).
+Null, unavailable, local, malformed and episode entries are skipped. Truncation is
+shown explicitly. Album tracks inherit the album artwork/name. Each match has its
+own error state; an osu! failure cannot erase other completed tracks. The enhanced
+form shows a pending state and renders the complete batch; native form submission
+also works without JavaScript. Filters require JavaScript.
+
+Spotify playlist availability depends on application quota/access. Both legacy
+`tracks/track` and newer `items/item` payloads are supported. Spotify's
+[2026 development-mode changes](https://developer.spotify.com/documentation/web-api/tutorials/february-2026-migration-guide)
+restrict playlist access; client credentials cannot unlock private/user-authorized
+playlists. Such responses produce an access error. This project intentionally does
+not add user authentication.
+
+Matching is pure and testable in `src/lib/map-finder/matching.ts`: Unicode and
+punctuation normalization, diacritic removal, common annotation cleanup, title and
+artist token overlap, Unicode metadata alternatives, and version penalties. Title
+contributes 65 points and artist 35. High confidence starts at 88, medium at 72;
+weak artist evidence is capped below medium, and a different version costs 24.
+These are heuristics, not probabilities. Candidates below 40 are omitted; up to six
+are sorted and deduplicated by beatmapset ID. Low-confidence candidates are collapsed.
+Search first uses artist + title, then one cleaned-title fallback if no medium/high
+match exists. Duration is omitted because osu! map length may reflect a song cut.
+
+Tokens are reused until 30 seconds before expiry with a shared refresh promise.
+Matching results cache for five minutes (maximum 500 entries), including no-match
+results, and identical pending searches are coalesced. Three concurrent osu!
+matching jobs are allowed per process. The site allows one active search per IP,
+up to six active searches globally, and 12 budget units per IP per 10 minutes:
+a single track costs one, a playlist/album six. Expired records are cleaned on
+requests; the limiter has a 10,000-record ceiling. It uses `getClientAddress()`,
+never arbitrary forwarded headers; unsupported adapters share a conservative key.
+Configure adapter-node's trusted proxy/client-address settings correctly at deployment.
+
+Requests have a 12-second timeout and at most two retries for network errors,
+429s or server errors. Backoff includes jitter. `Retry-After` is honored; delays
+above four seconds return a helpful error and retain a provider cooldown rather
+than holding the request open. All caches, cooldowns, concurrency and rate limits
+are per process and reset on restart; multiple instances do not share them.
+
+Local commands (npm scripts and existing tooling; no new dependencies):
+
+```sh
+npm install
+npm run dev
+npm run test:unit -- --run
+npm run test:integration
+npm run check
+npm run lint
+npm run build
+```
+
+Finder unit tests mock upstream APIs. Browser tests cover initial layout and
+validation with and without enhancement, and do not call Spotify or osu!.
+
 ## Language learner
 
 `/projects/language-learner` teaches Dutch and Korean (explained in English) and
@@ -32,3 +112,361 @@ It runs entirely in the browser: no database tables, migrations or API keys.
 - Progress, streaks and settings are stored in `localStorage`
   (`language-learner-v1`), per browser. Export/import on the overview page moves
   them between devices.
+
+## Shared accounts and permissions
+
+`/login`, `/register`, and `/account` are the central account pages. Better Auth
+handles email/password credentials and database-backed sessions for the entire
+site. Google and osu! buttons appear when the corresponding client ID and secret
+are configured. The header exposes account access everywhere.
+
+Copy the auth settings from `.env.example`. `BETTER_AUTH_SECRET` must be at least
+32 random characters; there is no runtime fallback secret. Set `ORIGIN` and
+`BETTER_AUTH_URL` to the same public origin (for example,
+`https://dev.stanrunge.dev` behind the local proxy). Register these exact callbacks:
+
+- Google: `<BETTER_AUTH_URL>/api/auth/callback/google`
+- osu!: `<BETTER_AUTH_URL>/api/auth/callback/osu`
+
+osu! uses the `identify` scope and a stable numeric account ID. Its API does not
+supply an email address, so these accounts use a reserved, unverified
+`<id>@osu.invalid` internal identifier, hidden from the account UI. Credential
+registration cannot claim that namespace. Provider accounts are linked explicitly
+from `/account` while logged in; there is no automatic merging by email. Link a
+provider **before** using it to create a separate account if you want shared
+progress. Existing separate accounts are not automatically merged.
+
+Every new user has role `user`. The shared server guards enforce administrator
+access for blog drafts/writes, list creation/editing, Animaldex moderation, and
+Progress edits. Legacy `ADMIN_SECRET`, `AUTH_TOKEN`, `admin_token`, and `token`
+logins no longer authorize anything. To grant yourself admin access, register,
+copy the account ID shown on `/account`, then run on the server:
+
+```sh
+npm run auth:role -- <user-id> admin
+# To revoke access:
+npm run auth:role -- <user-id> user
+```
+
+Role changes take effect on the next request; roles cannot be set through signup
+or profile updates. Public registrations are never made admin automatically.
+Future projects should reference the shared `user.id`, use `requireUser(event)`,
+and scope **both reads and writes** to the session user's ID. Use
+`requireAdmin(event)` for site-wide administration; never accept an owner ID from
+a form or infer permission from the presence of a cookie.
+
+Email/password registration works without a mail service. Email verification and
+password recovery emails are **not configured** yet. Connect a transactional mail
+sender and Better Auth's verification/reset hooks before offering those flows.
+Better Auth's rate limiting is enabled and currently uses per-process memory;
+a multi-instance deployment needs shared rate-limit storage. Account pages and
+saved progress require a working database. Authentication forms require JavaScript.
+
+## Game tracker
+
+`/projects/game-tracker` is the hub for completionist progress. Catchemall lives at
+`/projects/game-tracker/pokemon` (the old `/projects/catch-em-all` URL redirects),
+with location pages under `/projects/game-tracker/pokemon/locations`. Call of Duty
+games live at `/projects/game-tracker/call-of-duty/<slug>`. All catalog data is
+owned by the site: it is checked in under `data/` and seeded into Postgres, and
+nothing calls PokéAPI or Steam at runtime.
+
+### Pokémon catalog
+
+`npm run catalog:build` rebuilds two files from the pinned PokéAPI CSV snapshot
+(revision recorded inside them):
+
+- `data/pokemon-catalog.json`: **1,025 species, 53 game versions/expansions and
+  27,267 species/game entries**, plus each species' evolution chain and the
+  evolution method that applies **in each game** (Magnezone needs Mt. Coronet in
+  Platinum but a Thunder Stone in Sword).
+- `data/pokemon-locations.json`: 1,104 locations and about 67,000 encounters
+  (area, method, levels, rarity, form and conditions such as time of day, season
+  or swarm). Max Raid dens are folded per location.
+
+Each entry has one availability:
+
+- `catchable`: a documented encounter (wild, gift, static, in-game trade).
+- `evolve`: a pre-evolution is obtainable in that game and the method exists
+  there, e.g. Blastoise in Red is “Evolve Wartortle: Level 36+”.
+- `breed`: an obtainable relative can breed it (not in Gen 1, Let’s Go,
+  Colosseum/XD or the Legends games); incense babies name their incense.
+- `event_only`: a mythical in the game with no other route.
+- `unknown`: in the game's Pokédex data, but no confirmed route.
+
+`catchable`, `evolve` and `breed` can be ticked and count toward progress.
+PokéAPI has **no encounter data** for Brilliant Diamond/Shining Pearl, Legends:
+Arceus, Scarlet/Violet and their DLC, or Legends: Z-A, so those games stay
+`unknown` until another source is added. Missing pairs are not proof of absence.
+
+Administrators can correct any entry from the Pokémon detail dialog and add notes
+and a picture (JPEG/PNG/WebP, 5 MB, stored in R2 under `locations/`) on any
+location page. Rows carry a `source`: seeding refreshes `pokeapi` and `derived`
+rows and never touches `manual` ones. Location notes and pictures always survive
+reseeding, and user progress is keyed by entry, so it survives too.
+
+### Call of Duty catalog
+
+`data/call-of-duty.json` covers Black Ops, Black Ops II, Ghosts, Advanced
+Warfare, Black Ops III, Infinite Warfare and WWII. Weapon classes, post-launch
+(DLC/update) weapons and camo rules are **hand-maintained** from the Call of Duty
+Wiki (CC BY-SA). `npm run catalog:cod` refreshes only the achievement lists from
+the public Steam Community pages. Users tick per-weapon milestones (bought or
+unlocked, max level, Gold, Gold Cheetah); class and overall camos (Diamond, Dark
+Matter, Solar, Black Sky, Chrome, Chrome Tiger) are derived in
+`src/lib/game-tracker/mastery.ts`, and post-launch weapons never gate them.
+Weapon progress is keyed by a slug of the weapon name, so renaming a weapon in
+the JSON starts its progress over; the seed keeps removed items that already have
+progress and warns about them.
+
+### Database
+
+For an existing database using the current domain schemas:
+
+```sh
+npm run db:migrate:auth
+npm run db:migrate:foundation
+npm run db:migrate:money-tracker
+npm run db:migrate:game-tracker
+npm run db:seed
+npm run db:seed:cod
+```
+
+For a fresh database, run `npm run db:push` followed by both seeds.
+The migrations are additive and repeatable; they do not reset tables. The
+game-tracker migration adds the `evolve`/`breed` availability values, the entry
+`source` column (marking pre-existing hand-edited notes as `manual`), and the
+location, encounter, evolution and `tracker_*` tables. Unknown legacy OAuth
+providers cause the auth migration to fail for explicit review instead of
+guessing an issuer. Back up a production database before any schema migration.
+The scripts use `DB_URL` from `.env` via Bun.
+
+## Money tracker
+
+`/projects/money-tracker` is restricted to administrator accounts. Its tables
+are created by `npm run db:migrate:money-tracker`; apply that migration to each
+environment before deploying the new route. The deployment preflight checks for
+all nine tracker tables and the projection columns.
+
+Transactions can be entered manually or imported from a CSV or tab-separated
+export with `date,account,description,amount` headers. Dates accept
+`YYYY-MM-DD`, `DD/MM/YYYY`, or `Fri 1 Aug 2025`; amounts accept signed euros
+such as `-€1,234.50`. Quoted fields and comma, semicolon, or tab delimiters
+are supported, and empty rows are skipped. Imported entries are stored in date
+order. `Start balance` rows are imported as ordinary transactions, so leave
+the account opening balance at zero for those accounts. Import creates
+missing accounts and adds every row in one database transaction. It does not
+deduplicate repeat uploads, so import each file once. Accounts have an editable
+opening balance for money held before the first recorded transaction.
+The current balance and history stop at today; future dated transactions feed the
+wealth projection. The projection invests spendable money above the configured
+reserve in a modeled balance, even without an existing investment account.
+An existing investment account can optionally be selected as the starting
+balance for growth. The projection treats recurring entries as monthly
+equivalents from next month and can disable them to avoid double counting entries
+already recorded as future transactions. Its reserve, hypothetical return,
+contribution rule, and end date are editable. Wish list priority runs from tier 5
+(highest) to tier 1 (lowest), with cumulative estimated costs on the overview.
+
+The utility comparison uses meter readings within the configured billing period,
+the current per-unit rate, and prorated monthly advances. It excludes fixed
+charges, taxes, and past rate changes. Set the billing start to a recorded opening
+meter reading for a meaningful annual comparison.
+
+Seeding runs in one transaction and is repeatable. It preserves row IDs, existing
+catches, selected games, and hand-maintained availability notes. Pokémon are keyed
+by national number, games by PokéAPI version ID, and entries by Pokémon/game pair.
+Existing games with exactly matching names are adopted to preserve their IDs.
+The old broken reset and auth-schema overwrite commands have been removed.
+
+Availability is deliberately conservative:
+
+- `catchable`: PokéAPI supplies an encounter record. Conditions or a particular
+  form may be required. These entries contribute to the catch checklist.
+- `unknown` (`?`): a Pokédex or game-index record exists, but the acquisition method
+  has not been confirmed. Membership alone never becomes “catchable”.
+- A dash: no source record; this does **not** prove the Pokémon is unavailable.
+- Manually maintained trade/transfer/event entries remain supported.
+
+Encounter coverage varies by game and is especially incomplete for newer titles.
+The import does not invent evolution, breeding, gift, trade, transfer, or event
+rules from missing data. The page explains these limits. Catalog refreshes are
+explicit: update `REVISION` in `scripts/build-pokemon-catalog.py`, run
+`npm run catalog:build`, review the snapshot, and rerun the seed. Source:
+[PokéAPI data](https://github.com/PokeAPI/pokeapi/tree/master/data/v2/csv).
+
+Catches and selected games are stored server-side under the authenticated user's
+ID. Repeated catch requests are idempotent, writes validate catalog IDs, and an
+explicit empty game selection survives reloads. The UI rolls back failed saves
+and prevents overlapping changes to the same catch.
+
+`npm run test:unit -- --run` covers authorization, safe return paths, osu! identity
+mapping, and catalog integrity alongside existing tests. `npm run test:auth` runs
+browser/database checks against a migrated, seeded **development** database; it
+creates temporary test accounts/content and removes them afterward. It checks
+registration, login/logout, cookie replay, role escalation attempts, cross-user
+progress isolation, admin routes, game selection, and OAuth authorization startup.
+Completing Google/osu! consent still requires real provider credentials and the
+registered callback URL. The integration follows the installed Better Auth APIs;
+see [SvelteKit integration](https://better-auth.com/docs/integrations/svelte-kit)
+and [Generic OAuth](https://better-auth.com/docs/plugins/generic-oauth).
+
+## Staging and production rollout
+
+Use `staging` as the working branch and set it as the repository's default branch
+in Gitea after the first push. A direct push to `staging` runs unit tests, builds
+the site and a SHA-tagged Docker image, and deploys that image to
+`https://staging.stanrunge.dev`. After checking staging, use Gitea Actions →
+**Deploy production** → **Run workflow**. That action retests the current staging
+commit, verifies that staging runs its exact image, and starts the **same image**
+at `https://stanrunge.dev`. It does not rebuild the image or need a PR. Pushes to
+`main` no longer deploy production; once `staging` is the default branch, its
+current manual workflow is also the one shown in Gitea's Actions UI.
+
+The staging Docker build runs `bun install --frozen-lockfile`, unit tests, and
+the production build, so a failed test never produces a deployable image. The
+production button verifies the exact tested staging image before switching
+production. Browser/database tests need a
+dedicated test database and are not part of this gate yet. `npm run check` and
+full lint are also not gates while the repository has existing unrelated
+diagnostics; add them after those are fixed. A failed test, build, or preflight
+prevents the switch; a failed post-deploy health check attempts to restore the
+previous image tag. Deployment never migrates or seeds either database. The
+Gitea runner needs Docker CLI/socket access and the existing `DEPLOY_KEY` secret.
+
+Before either push, configure the existing private environment file on that
+host. Staging uses `/etc/personal-website-staging/staging.env`; production uses
+`/etc/personal-website/prod.env`. Each file needs `DB_URL`, a unique random
+`BETTER_AUTH_SECRET` of at least 32 characters, and both `ORIGIN` and
+`BETTER_AUTH_URL` set exactly to its public HTTPS origin. Add the Google and
+osu! client ID/secret pairs for whichever sign-in buttons should be enabled.
+List and Animaldex picture uploads reuse the existing R2 bucket; set the five
+`R2_*` values from `.env.example`. The app server uploads images to R2, so the
+bucket does not need browser CORS rules for these forms. Each image is limited
+to 5 MB; the container sets `BODY_SIZE_LIMIT=6M` for multipart requests.
+Their provider console callback URLs must use that environment's origin:
+
+```text
+https://staging.stanrunge.dev/api/auth/callback/google
+https://staging.stanrunge.dev/api/auth/callback/osu
+https://stanrunge.dev/api/auth/callback/google
+https://stanrunge.dev/api/auth/callback/osu
+```
+
+For this first rollout, review the SQL and back up each database. The first push
+to `staging` may pass **Transfer tested image** but fail **Deploy staging** at
+the database preflight. That is expected if the database has not been migrated:
+the image is then available on the deployment host for you to run the migrations
+yourself. If `animals`, `sightings`, and `photos` still have UUID IDs, first apply
+`scripts/animaldex-uuid-to-serial.sql` with `psql` once. If the database has the
+legacy singular tables `game`, `pokemon_game_entry`, `user_entry`, and `user_game`,
+then apply `scripts/legacy-schema-migration.sql` with `psql` once. That migration
+renames those tables without discarding their rows and creates the empty base
+`lists` and `list_items` tables. Run both reviewed SQL files before the container
+commands below. On the deployment host, use the SHA from the staging Actions run:
+
+```sh
+release_image=git.stanrunge.dev/stan/personal-website:<staging-sha>
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-auth-catalog.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-foundation.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-money-tracker.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-game-tracker.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-tracker-journal.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-catalog.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-cod-catalog.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-pokemon-research.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-cod-challenges.ts
+sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-cod-art.ts
+```
+
+These catalog seeds use the reviewed files in `data/` and upsert catalog rows;
+they preserve user accounts, catches, ranks, and saved challenge progress. Do not
+copy or restore the entire development database over staging or production.
+Repeat the same migration and seed commands for staging first, replacing the
+environment file with `/etc/personal-website/prod.env` only after staging has
+passed its preflight and been reviewed. Back up production before applying its
+migrations. Rerun the same staging Actions run after the staging migration and
+seed complete. Click the production button only after production migration and
+seed complete. Future releases without database changes need only a staging push
+and the production button.
+
+The foundation migration gives lists slugs, progress, pictures, and blog tags;
+gives site-owned join/settings rows numeric IDs; and adds `created_at`/
+`updated_at` with a database update trigger. Blog posts already had numeric IDs
+and timestamps. Better Auth's account/session IDs stay as text because they are
+provider-managed identities. The deploy script verifies the migrations and
+catalog before switching the image, then checks `/login` and Catchemall. A
+failed app health check attempts to restore the previous image tag. Database
+changes are not rolled back with the app image.
+
+The release image contains `scripts/`, `data/`, and the schema files needed by
+the migration tools. After registration in each environment, promote your own
+account using the host's matching environment file, for example:
+
+```sh
+sudo docker run --rm --network backend \
+  --env-file /etc/personal-website-staging/staging.env \
+  git.stanrunge.dev/stan/personal-website:<deployed-sha> \
+  bun scripts/set-admin.ts <account-id> admin
+```
+
+Use `/etc/personal-website/prod.env` and its deployed image SHA for production.
+The account ID is shown on `/account`. No public signup grants admin privileges.
+
+# CS prices
+
+`/projects/cs-prices` reads only from Postgres. Each Steam market hash name is an
+individual listing. The separate `cs_item_dimensions` table makes it possible to
+compare or group by weapon type, weapon name, weapon plus skin, skin, wear, StatTrak, Souvenir, and
+additional imported dimensions such as collection or rarity. Group chart points
+average the available per-listing prices equally within each time bucket. Raw
+observations retain their source, currency, timestamp, kind, price in cents, and
+volume. Chart responses use up to roughly 240 time buckets per series and allow
+up to 1,000 selected series; actual response time still depends on database size
+and hardware.
+
+With `DB_URL` pointed at a running Postgres database:
+
+```sh
+npm run db:migrate:cs-prices
+npm run cs-prices:ingest -- catalog
+STEAM_SESSIONID=... STEAM_LOGIN_SECURE=... npm run cs-prices:ingest -- history
+```
+
+The catalog pass pages through Counter-Strike market listings and saves a current
+lowest-listing snapshot. The history pass visits each catalog item and stores
+Steam's available historical sale-price and volume points. It needs a Steam web
+session, may take many hours for the full catalog, and can be resumed with
+`--start=<item offset>` and bounded with `--limit=<items>`. Requests pause for two
+seconds by default (`--delay-ms=...`, minimum one second), retry temporary
+failures, and upsert observations safely. Run catalog regularly for new listings
+and snapshots; run history regularly for new sale points. Steam's market web
+endpoints are undocumented and may change or limit access. The historical
+granularity and completeness are bounded by what Steam returns; running this
+collector cannot reconstruct older high-frequency prices that Steam no longer
+exposes.
+
+For another provider or a prepared archive, import JSON Lines instead:
+
+```sh
+npm run cs-prices:ingest -- import ./prices.jsonl
+```
+
+Each line needs `marketHashName`, `source`, `currency` (three-letter ISO code),
+`observedAt` (ISO timestamp), `priceCents`, and `kind`. Optional `name`, `volume`,
+and `dimensions` add detail. For example:
+
+```json
+{
+	"marketHashName": "AK-47 | Redline (Field-Tested)",
+	"source": "archive",
+	"currency": "USD",
+	"observedAt": "2026-09-24T12:00:00Z",
+	"priceCents": 1850,
+	"volume": 42,
+	"kind": "sale-median",
+	"dimensions": { "collection": "Phoenix", "rarity": "classified" }
+}
+```
+
+Imported sources and price kinds appear in the page's source and kind controls.