stan/website
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases Wiki Activity

deploy: prod-only pipeline with Drizzle migrations; dev refresh + preview

Browse Source 
- push to main builds, backs up prod (pg_dump 17), runs db-migrate + seeds,
  preflights and deploys; rollback = workflow_dispatch with rollback_tag
- staging workflow, promote step and migrate-*.ts / old SQL removed
- DB_URL = app role, DB_MIGRATE_URL = owner role
- scripts/db-refresh-dev.sh (prod dump -> local dev DB), scripts/preview.sh <branch>
This commit is contained in:
Stan Runge
2026-10-23 22:03:42 +00:00
parent 4cfe6bad6c
commit 2915135ed8
30 changed files with 425 additions and 1114 deletions
Download Patch File Download Diff File Expand all files Collapse all files

183
README.md
View File

@@ -227,25 +227,14 @@ progress and warns about them.
### Database
For an existing database using the current domain schemas:
```sh
npm run db:migrate:auth
npm run db:migrate:foundation
npm run db:migrate:money-tracker
npm run db:migrate:game-tracker
npm run db:seed
npm run db:seed:cod
```
For a fresh database, run `npm run db:push` followed by both seeds.
The migrations are additive and repeatable; they do not reset tables. The
game-tracker migration adds the `evolve`/`breed` availability values, the entry
`source` column (marking pre-existing hand-edited notes as `manual`), and the
location, encounter, evolution and `tracker_*` tables. Unknown legacy OAuth
providers cause the auth migration to fail for explicit review instead of
guessing an issuer. Back up a production database before any schema migration.
The scripts use `DB_URL` from `.env` via Bun.
The schema is managed by Drizzle (`src/lib/server/db/schema/`, migrations in
`drizzle/`). Postgres schemas: `auth`, `money`, `house`, `ref`, `games`, `cs`,
`animaldex`, `site`, `meta` (Drizzle-managed) and `lists` (one real table per list,
created at runtime through the `meta.create_list` family of SQL functions, so it is
excluded from Drizzle via `schemaFilter`). See [Development](#development) for the
local database and migrations, and `db/reorg/README.md` for the one-time move of the
old flat schema to this layout. Catalog seeds (`npm run db:seed`, `db:seed:cod`, …)
upsert reference data and run on every production deploy.
## Money tracker
@@ -317,112 +306,87 @@ registered callback URL. The integration follows the installed Better Auth APIs;
see [SvelteKit integration](https://better-auth.com/docs/integrations/svelte-kit)
and [Generic OAuth](https://better-auth.com/docs/plugins/generic-oauth).
## Staging and production rollout
## Development
Use `staging` as the working branch and set it as the repository's default branch
in Gitea after the first push. A direct push to `staging` runs unit tests, builds
the site and a SHA-tagged Docker image, and deploys that image to
`https://staging.stanrunge.dev`. After checking staging, use Gitea Actions →
**Deploy production** → **Run workflow**. That action retests the current staging
commit, verifies that staging runs its exact image, and starts the **same image**
at `https://stanrunge.dev`. It does not rebuild the image or need a PR. Pushes to
`main` no longer deploy production; once `staging` is the default branch, its
current manual workflow is also the one shown in Gitea's Actions UI.
There are two environments: local dev and production. Prod is the source of truth
for data; the dev database is a disposable copy of it.
The staging Docker build runs `bun install --frozen-lockfile`, unit tests, and
the production build, so a failed test never produces a deployable image. The
production button verifies the exact tested staging image before switching
production. Browser/database tests need a
dedicated test database and are not part of this gate yet. `npm run check` and
full lint are also not gates while the repository has existing unrelated
diagnostics; add them after those are fixed. A failed test, build, or preflight
prevents the switch; a failed post-deploy health check attempts to restore the
previous image tag. Deployment never migrates or seeds either database. The
Gitea runner needs Docker CLI/socket access and the existing `DEPLOY_KEY` secret.
```sh
cp .env.example .env # DB_URL (app role) + DB_MIGRATE_URL (owner role) -> local personal_dev
scripts/db-refresh-dev.sh # fresh prod dump -> personal_dev (sessions cleared)
bun run db:migrate # apply this branch's pending Drizzle migrations
bun run dev
```
Before either push, configure the existing private environment file on that
host. Staging uses `/etc/personal-website-staging/staging.env`; production uses
`/etc/personal-website/prod.env`. Each file needs `DB_URL`, a unique random
`BETTER_AUTH_SECRET` of at least 32 characters, and both `ORIGIN` and
`BETTER_AUTH_URL` set exactly to its public HTTPS origin. Add the Google and
osu! client ID/secret pairs for whichever sign-in buttons should be enabled.
List and Animaldex picture uploads reuse the existing R2 bucket; set the five
`R2_*` values from `.env.example`. The app server uploads images to R2, so the
bucket does not need browser CORS rules for these forms. Each image is limited
to 5 MB; the container sets `BODY_SIZE_LIMIT=6M` for multipart requests.
Their provider console callback URLs must use that environment's origin:
`db-refresh-dev.sh` reads `PROD_DB_URL` (a role that can read everything, e.g.
`personal`) and `DEV_ADMIN_URL` (superuser on the dev cluster) from the environment
or `.env.dev-refresh` (git-ignored, never commit it). It refuses to overwrite the
source database or any database named `personal`. `PG_DOCKER=1` uses the
`postgres:17-alpine` image when local `pg_dump` 17 isn't installed.
Schema changes: edit `src/lib/server/db/schema/`, run `bun run db:generate --name
<change>`, review the SQL in `drizzle/`, then `bun run db:migrate`. Never edit an
applied migration; never use `db:push` against prod.
`scripts/preview.sh <branch> [port]` runs a branch against a throwaway copy of the
dev database (migrated with that branch's migrations) and cleans everything up on
Ctrl-C.
### Database roles
| env var | role | used for |
| ---------------- | -------------- | --------------------------------------------------------------- |
| `DB_URL` | `personal_app` | the app at runtime: DML only, calls the `meta.*` list functions |
| `DB_MIGRATE_URL` | `personal` | Drizzle migrations, seeds, backups (`pg_dump`) |
`personal_ddl` owns the `lists` schema (only reachable through the SECURITY DEFINER
`meta.*` functions); `personal_ro` is read-only. The deploy preflight fails if
`DB_URL` can create objects, i.e. if it isn't the app role.
## Production
Push or merge to `main` → Gitea Actions (`.gitea/workflows/deploy.yml`) builds a
SHA-tagged image (unit tests + production build run inside the Docker build),
transfers it, and runs `scripts/deploy-remote.sh <sha> deploy` on the host:
1. checks the image and that `DB_MIGRATE_URL` is set in `/etc/personal-website/prod.env`;
2. `pg_dump` of prod (postgres:17-alpine) into the backup directory;
3. `bun scripts/db-migrate.ts` (Drizzle migrations) and the catalog seeds;
4. preflight (tables, column types, app-role privileges), switch the container, health checks.
A failed health check restores the previous image tag automatically. Manual
rollback: Gitea Actions → **Deploy production** → **Run workflow** with
`rollback_tag=<previous sha>`; this redeploys that image without migrating or
seeding. Database changes are not rolled back with the image; restore the
pre-deploy dump if a migration has to be undone. Only `main` deploys; other
branches never trigger the workflow. The runner needs Docker CLI/socket access
and the `DEPLOY_KEY` secret.
`/etc/personal-website/prod.env` needs `DB_URL`, `DB_MIGRATE_URL`, a unique random
`BETTER_AUTH_SECRET` (32+ characters), and `ORIGIN`/`BETTER_AUTH_URL` set to
`https://stanrunge.dev`, plus the Google and osu! client ID/secret pairs and the five
`R2_*` values from `.env.example`. Picture uploads (lists, Animaldex) go through the
app server to R2 (5 MB per image; the container sets `BODY_SIZE_LIMIT=6M`). OAuth
callback URLs:
```text
https://staging.stanrunge.dev/api/auth/callback/google
https://staging.stanrunge.dev/api/auth/callback/osu
https://stanrunge.dev/api/auth/callback/google
https://stanrunge.dev/api/auth/callback/osu
```
For this first rollout, review the SQL and back up each database. The first push
to `staging` may pass **Transfer tested image** but fail **Deploy staging** at
the database preflight. That is expected if the database has not been migrated:
the image is then available on the deployment host for you to run the migrations
yourself. If `animals`, `sightings`, and `photos` still have UUID IDs, first apply
`scripts/animaldex-uuid-to-serial.sql` with `psql` once. If the database has the
legacy singular tables `game`, `pokemon_game_entry`, `user_entry`, and `user_game`,
then apply `scripts/legacy-schema-migration.sql` with `psql` once. That migration
renames those tables without discarding their rows and creates the empty base
`lists` and `list_items` tables. Run both reviewed SQL files before the container
commands below. On the deployment host, use the SHA from the staging Actions run:
After registering, promote your account (ID shown on `/account`):
```sh
release_image=git.stanrunge.dev/stan/personal-website:<staging-sha>
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-auth-catalog.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-foundation.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-money-tracker.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-game-tracker.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/migrate-tracker-journal.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-catalog.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-cod-catalog.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-pokemon-research.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-cod-challenges.ts
sudo docker run --rm --network backend --env-file /etc/personal-website-staging/staging.env "$release_image" bun scripts/seed-cod-art.ts
sudo docker run --rm --network backend --env-file /etc/personal-website/prod.env \
git.stanrunge.dev/stan/personal-website:<deployed-sha> bun scripts/set-admin.ts <account-id> admin
```
These catalog seeds use the reviewed files in `data/` and upsert catalog rows;
they preserve user accounts, catches, ranks, and saved challenge progress. Do not
copy or restore the entire development database over staging or production.
Repeat the same migration and seed commands for staging first, replacing the
environment file with `/etc/personal-website/prod.env` only after staging has
passed its preflight and been reviewed. Back up production before applying its
migrations. Rerun the same staging Actions run after the staging migration and
seed complete. Click the production button only after production migration and
seed complete. Future releases without database changes need only a staging push
and the production button.
The foundation migration gives lists slugs, progress, pictures, and blog tags;
gives site-owned join/settings rows numeric IDs; and adds `created_at`/
`updated_at` with a database update trigger. Blog posts already had numeric IDs
and timestamps. Better Auth's account/session IDs stay as text because they are
provider-managed identities. The deploy script verifies the migrations and
catalog before switching the image, then checks `/login` and Catchemall. A
failed app health check attempts to restore the previous image tag. Database
changes are not rolled back with the app image.
The release image contains `scripts/`, `data/`, and the schema files needed by
the migration tools. After registration in each environment, promote your own
account using the host's matching environment file, for example:
```sh
sudo docker run --rm --network backend \
--env-file /etc/personal-website-staging/staging.env \
git.stanrunge.dev/stan/personal-website:<deployed-sha> \
bun scripts/set-admin.ts <account-id> admin
```
Use `/etc/personal-website/prod.env` and its deployed image SHA for production.
The account ID is shown on `/account`. No public signup grants admin privileges.
No public signup grants admin privileges.
# CS prices
`/projects/cs-prices` reads only from Postgres. Each Steam market hash name is an
individual listing. The separate `cs_item_dimensions` table makes it possible to
individual listing. The separate `cs.item_dimensions` table makes it possible to
compare or group by weapon type, weapon name, weapon plus skin, skin, wear, StatTrak, Souvenir, and
additional imported dimensions such as collection or rarity. Group chart points
average the available per-listing prices equally within each time bucket. Raw
@@ -431,10 +395,9 @@ volume. Chart responses use up to roughly 240 time buckets per series and allow
up to 1,000 selected series; actual response time still depends on database size
and hardware.
With `DB_URL` pointed at a running Postgres database:
With `DB_URL` pointed at a migrated database:
```sh
npm run db:migrate:cs-prices
npm run cs-prices:ingest -- catalog
STEAM_SESSIONID=... STEAM_LOGIN_SECURE=... npm run cs-prices:ingest -- history
```