music: self-hosted scrobbler (Spotify poller, Last.fm + streaming-history importers, /trackers/music, homepage listening card)
All checks were successful
Deploy production / deploy (push) Successful in 2m37s

- music schema: artists/albums/tracks (+album art), plays with source + dedupe, now_playing,
  spotify_auth, import_state; plays_deduped view hides Last.fm scrobbles Spotify also has
- in-app Spotify poller (60s, leased) started from hooks init; no new role or env needed
- /api/spotify/connect + callback (owner only), /trackers/music public stats
- scripts: music-poll, music-import-lastfm (resumable), music-import-history
- deploy/music-collector: optional host timer with its own role
This commit is contained in:
2026-10-24 15:56:50 +00:00
parent 45c124a365
commit b757269151
37 changed files with 22124 additions and 6 deletions

View File

@@ -20,6 +20,8 @@ COPY --from=builder /app/src/lib/cs-prices ./src/lib/cs-prices
# scripts/house-listing.ts (manual listings CLI) imports these pure helpers.
COPY --from=builder /app/src/lib/house-tracker ./src/lib/house-tracker
COPY --from=builder /app/src/lib/availability.ts ./src/lib/availability.ts
# scripts/music-*.ts (Spotify poll, Last.fm and streaming-history imports) import these.
COPY --from=builder /app/src/lib/music ./src/lib/music
COPY package.json ./
EXPOSE 3000
ENV NODE_ENV=production

View File

@@ -0,0 +1,66 @@
# music (self-hosted Last.fm)
`/trackers/music` (stats) and the "listening now" card on the homepage read the `music`
schema in the `personal` database:
| table / view | contents |
| ------------------------- | ----------------------------------------------------------------------------------------------- |
| `artists`, `albums` | Spotify ids when known (imports are matched by name), album art URLs (640 px + small) |
| `tracks`, `track_artists` | `match_key` = normalised `artist\|title` (`src/lib/music/normalize.ts`) for cross-source dedupe |
| `plays` | one row per play: `source` spotify / lastfm / youtube / local, `origin` api / export / import |
| `plays_deduped` (view) | `plays` minus Last.fm scrobbles that Spotify also has (what every stat reads) |
| `now_playing` | row 1, refreshed every minute |
| `spotify_auth` | row 1, refresh token + poller state. **Not** granted to `personal_ro` |
| `import_state` | resumable backfill progress (`lastfm:<user>`) |
## Spotify (live)
1. In the Spotify developer dashboard (the existing app, `SPOTIFY_CLIENT_ID`) add the redirect
URIs `https://stanrunge.dev/api/spotify/callback` and
`https://dev.stanrunge.dev/api/spotify/callback`.
2. Logged in as the owner, open <https://stanrunge.dev/api/spotify/connect> once. Scopes:
`user-read-currently-playing user-read-recently-played user-read-playback-state user-top-read`.
3. That's it: the **app itself** polls every minute (`src/lib/server/music/poller.ts`, started
from `hooks.server.ts` `init`) with its own role and Spotify credentials. Status (last poll,
last error) is shown to the owner on `/trackers/music`.
### Optional: poll from the host instead
Only if the poller should keep running independent of the app container. Needs its own role:
```sh
sudo docker exec -i postgres-db-1 psql -U postgres -d personal <<'SQL'
CREATE ROLE music_collector LOGIN CONNECTION LIMIT 3 PASSWORD '<pick one>';
GRANT CONNECT ON DATABASE personal TO music_collector;
GRANT USAGE ON SCHEMA music TO music_collector;
GRANT SELECT, INSERT, UPDATE ON music.artists, music.albums, music.tracks, music.track_artists,
music.plays, music.now_playing, music.import_state TO music_collector;
GRANT SELECT, UPDATE ON music.spotify_auth TO music_collector;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA music TO music_collector;
SQL
```
`~/.config/music-collector.env` (mode 600): `MUSIC_DB_URL=postgresql://music_collector:…@127.0.0.1:5440/personal`,
`SPOTIFY_CLIENT_ID=…`, `SPOTIFY_CLIENT_SECRET=…`. Then `bash deploy/music-collector/install.sh --enable`
and set `MUSIC_POLLER=off` in prod.env. (The music migration grants the same when the role
already exists.) Debug: `journalctl --user -u music-collector -n 20`.
## Backfills
Both run from the deployed image with the app role, so no extra credentials:
```sh
tag=$(sudo sed -n 's/^IMAGE_TAG=//p' /opt/personal-website/.env)
# Last.fm (needs an API key from https://www.last.fm/api/account/create); resumable, re-run any time
sudo docker run --rm --network backend --env-file /etc/personal-website/prod.env \
-e LASTFM_API_KEY=<key> git.stanrunge.dev/stan/website:$tag bun scripts/music-import-lastfm.ts --user stanrunge
# Spotify Extended Streaming History (unzipped export in ~/spotify-export)
sudo docker run --rm --network backend --env-file /etc/personal-website/prod.env \
-v ~/spotify-export:/data:ro git.stanrunge.dev/stan/website:$tag bun scripts/music-import-history.ts /data
```
Dedupe: within one source a play within 60 s of the same `match_key` is skipped on insert;
across sources `plays_deduped` hides a Last.fm scrobble when a Spotify play of the same
`match_key` ended between 3 min before and track length + 3 min after the scrobble time
(Last.fm records when a track started, Spotify when it ended). Listening time uses
`ms_played` when known, else the track length.

View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Install the optional host-side Spotify poller as a systemd user timer (run as stan on the
# server). Copies the units; enables the timer only with --enable (needs
# ~/.config/music-collector.env, see README.md). Normally not needed: the app polls itself.
set -euo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
install -D -m 644 "$here/music-collector.service" "$HOME/.config/systemd/user/music-collector.service"
install -D -m 644 "$here/music-collector.timer" "$HOME/.config/systemd/user/music-collector.timer"
systemctl --user daemon-reload
if [ "${1:-}" = --enable ]; then
test -f "$HOME/.config/music-collector.env" || { echo 'Missing ~/.config/music-collector.env' >&2; exit 1; }
systemctl --user enable --now music-collector.timer
systemctl --user start music-collector.service || true
systemctl --user --no-pager status music-collector.service | head -5
else
echo 'Installed (not enabled). Enable with: bash deploy/music-collector/install.sh --enable'
fi

View File

@@ -0,0 +1,15 @@
[Unit]
Description=Spotify scrobbler poll (now playing + recently played -> music.*) for stanrunge.dev/trackers/music
Wants=network-online.target
After=network-online.target
ConditionPathExists=%h/.config/music-collector.env
[Service]
Type=oneshot
# A checkout of stan/website on main. Not the live dev tree.
WorkingDirectory=%h/dev/website-music
# MUSIC_DB_URL (music_collector role), SPOTIFY_CLIENT_ID, SPOTIFY_CLIENT_SECRET; mode 600.
EnvironmentFile=%h/.config/music-collector.env
ExecStart=/usr/bin/bun scripts/music-poll.ts
TimeoutStartSec=50
Nice=10

View File

@@ -0,0 +1,10 @@
[Unit]
Description=Spotify scrobbler poll every minute (optional; the app polls by itself)
[Timer]
OnBootSec=45s
OnUnitActiveSec=60s
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@@ -25,7 +25,8 @@ export default defineConfig({
'tasks',
'groceries',
'home',
'cars'
'cars',
'music'
],
migrations: { schema: 'drizzle', table: '__drizzle_migrations' },
verbose: true,

View File

@@ -0,0 +1,159 @@
CREATE SCHEMA "music";
--> statement-breakpoint
CREATE TABLE "music"."albums" (
"id" serial PRIMARY KEY,
"spotify_id" text,
"name" text NOT NULL,
"artist_id" integer,
"image_url" text,
"image_url_small" text,
"release_date" text,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "music"."artists" (
"id" serial PRIMARY KEY,
"spotify_id" text,
"name" text NOT NULL,
"image_url" text,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "music"."import_state" (
"key" text PRIMARY KEY,
"data" jsonb NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "music"."now_playing" (
"id" smallint PRIMARY KEY DEFAULT 1,
"track_id" integer,
"is_playing" boolean DEFAULT false NOT NULL,
"progress_ms" integer,
"device_name" text,
"device_type" text,
"context_uri" text,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
CONSTRAINT "now_playing_single_row" CHECK ("id" = 1)
);
--> statement-breakpoint
CREATE TABLE "music"."plays" (
"id" serial PRIMARY KEY,
"track_id" integer NOT NULL,
"played_at" timestamp with time zone NOT NULL,
"source" text DEFAULT 'spotify' NOT NULL,
"origin" text DEFAULT 'api' NOT NULL,
"ms_played" integer,
"context_type" text,
"context_uri" text,
"platform" text,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
CONSTRAINT "plays_source_check" CHECK ("source" in ('spotify', 'lastfm', 'youtube', 'local')),
CONSTRAINT "plays_origin_check" CHECK ("origin" in ('api', 'export', 'import', 'manual'))
);
--> statement-breakpoint
CREATE TABLE "music"."spotify_auth" (
"id" smallint PRIMARY KEY DEFAULT 1,
"spotify_user_id" text NOT NULL,
"display_name" text,
"refresh_token" text NOT NULL,
"access_token" text,
"access_expires_at" timestamp with time zone,
"scope" text,
"recent_cursor" bigint,
"needs_reauth" boolean DEFAULT false NOT NULL,
"lease_until" timestamp with time zone,
"last_poll_at" timestamp with time zone,
"last_error" text,
"connected_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
CONSTRAINT "spotify_auth_single_row" CHECK ("id" = 1)
);
--> statement-breakpoint
CREATE TABLE "music"."track_artists" (
"track_id" integer,
"artist_id" integer,
"position" smallint DEFAULT 0 NOT NULL,
CONSTRAINT "track_artists_pkey" PRIMARY KEY("track_id","artist_id")
);
--> statement-breakpoint
CREATE TABLE "music"."tracks" (
"id" serial PRIMARY KEY,
"spotify_id" text,
"name" text NOT NULL,
"album_id" integer,
"artist_id" integer,
"duration_ms" integer,
"explicit" boolean,
"match_key" text NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE UNIQUE INDEX "albums_spotify_id_key" ON "music"."albums" ("spotify_id") WHERE "spotify_id" is not null;--> statement-breakpoint
CREATE INDEX "albums_name_idx" ON "music"."albums" (lower("name"));--> statement-breakpoint
CREATE UNIQUE INDEX "artists_spotify_id_key" ON "music"."artists" ("spotify_id") WHERE "spotify_id" is not null;--> statement-breakpoint
CREATE INDEX "artists_name_idx" ON "music"."artists" (lower("name"));--> statement-breakpoint
CREATE UNIQUE INDEX "plays_dedupe_key" ON "music"."plays" ("source","track_id","played_at");--> statement-breakpoint
CREATE INDEX "plays_played_at_idx" ON "music"."plays" ("played_at");--> statement-breakpoint
CREATE INDEX "plays_source_played_idx" ON "music"."plays" ("source","played_at");--> statement-breakpoint
CREATE INDEX "plays_track_played_idx" ON "music"."plays" ("track_id","played_at");--> statement-breakpoint
CREATE INDEX "track_artists_artist_idx" ON "music"."track_artists" ("artist_id");--> statement-breakpoint
CREATE UNIQUE INDEX "tracks_spotify_id_key" ON "music"."tracks" ("spotify_id") WHERE "spotify_id" is not null;--> statement-breakpoint
CREATE INDEX "tracks_name_idx" ON "music"."tracks" (lower("name"));--> statement-breakpoint
CREATE INDEX "tracks_match_key_idx" ON "music"."tracks" ("match_key");--> statement-breakpoint
ALTER TABLE "music"."albums" ADD CONSTRAINT "albums_artist_id_artists_id_fkey" FOREIGN KEY ("artist_id") REFERENCES "music"."artists"("id") ON DELETE SET NULL;--> statement-breakpoint
ALTER TABLE "music"."now_playing" ADD CONSTRAINT "now_playing_track_id_tracks_id_fkey" FOREIGN KEY ("track_id") REFERENCES "music"."tracks"("id") ON DELETE SET NULL;--> statement-breakpoint
ALTER TABLE "music"."plays" ADD CONSTRAINT "plays_track_id_tracks_id_fkey" FOREIGN KEY ("track_id") REFERENCES "music"."tracks"("id") ON DELETE CASCADE;--> statement-breakpoint
ALTER TABLE "music"."track_artists" ADD CONSTRAINT "track_artists_track_id_tracks_id_fkey" FOREIGN KEY ("track_id") REFERENCES "music"."tracks"("id") ON DELETE CASCADE;--> statement-breakpoint
ALTER TABLE "music"."track_artists" ADD CONSTRAINT "track_artists_artist_id_artists_id_fkey" FOREIGN KEY ("artist_id") REFERENCES "music"."artists"("id") ON DELETE CASCADE;--> statement-breakpoint
ALTER TABLE "music"."tracks" ADD CONSTRAINT "tracks_album_id_albums_id_fkey" FOREIGN KEY ("album_id") REFERENCES "music"."albums"("id") ON DELETE SET NULL;--> statement-breakpoint
ALTER TABLE "music"."tracks" ADD CONSTRAINT "tracks_artist_id_artists_id_fkey" FOREIGN KEY ("artist_id") REFERENCES "music"."artists"("id") ON DELETE SET NULL;--> statement-breakpoint
-- Hand-written below. Cross-source dedupe: every stat reads plays_deduped, which hides a
-- Last.fm scrobble when Spotify (live API or the streaming-history export) has the same
-- normalised artist+title around the same time. Last.fm stores when a track started,
-- Spotify when it ended, so the Spotify end may fall from 3 min before the scrobble to
-- track length + 3 min after it. Spotify rows always win; Last.fm fills the gaps.
CREATE VIEW "music"."plays_deduped" AS
SELECT p.* FROM "music"."plays" p
WHERE p.source <> 'lastfm' OR NOT EXISTS (
SELECT 1
FROM "music"."tracks" lt, "music"."plays" s
JOIN "music"."tracks" st ON st.id = s.track_id
WHERE lt.id = p.track_id
AND s.source = 'spotify'
AND s.played_at >= p.played_at - interval '3 minutes'
AND s.played_at <= p.played_at + interval '25 minutes'
AND s.played_at <= p.played_at + make_interval(secs => coalesce(st.duration_ms, 600000) / 1000.0) + interval '3 minutes'
AND st.match_key = lt.match_key
);
--> statement-breakpoint
-- Grants only run on prod, where migrations run as `personal`. Listening history is public
-- (personal_ro may read it); the Spotify refresh token is for the app role only.
REVOKE ALL ON SCHEMA music FROM PUBLIC;
--> statement-breakpoint
DO $$
BEGIN
IF current_user = 'personal' AND EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'personal_app') THEN
GRANT USAGE ON SCHEMA music TO personal_app;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA music TO personal_app;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA music TO personal_app;
ALTER DEFAULT PRIVILEGES FOR ROLE personal IN SCHEMA music
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO personal_app;
ALTER DEFAULT PRIVILEGES FOR ROLE personal IN SCHEMA music
GRANT USAGE, SELECT ON SEQUENCES TO personal_app;
END IF;
IF current_user = 'personal' AND EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'personal_ro') THEN
GRANT USAGE ON SCHEMA music TO personal_ro;
GRANT SELECT ON music.artists, music.albums, music.tracks, music.track_artists, music.plays,
music.plays_deduped, music.now_playing TO personal_ro;
END IF;
-- Optional host-side poller (deploy/music-collector). The role is created out of band
-- (it has a password); see deploy/music-collector/README.md.
IF current_user = 'personal' AND EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'music_collector') THEN
GRANT USAGE ON SCHEMA music TO music_collector;
GRANT SELECT, INSERT, UPDATE ON music.artists, music.albums, music.tracks, music.track_artists,
music.plays, music.now_playing, music.import_state TO music_collector;
GRANT SELECT, UPDATE ON music.spotify_auth TO music_collector;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA music TO music_collector;
END IF;
END $$;

File diff suppressed because it is too large Load Diff

16
scripts/music-db.ts Normal file
View File

@@ -0,0 +1,16 @@
// Shared database connection for the music scripts: MUSIC_DB_URL (e.g. the optional
// music_collector role) or DB_URL (the app role, as in the image's prod.env).
import postgres from 'postgres';
export function musicDb() {
const url = process.env.MUSIC_DB_URL || process.env.DB_URL;
if (!url) throw new Error('Set MUSIC_DB_URL or DB_URL');
return postgres(url, { max: 1, prepare: false, connect_timeout: 10, onnotice: () => {} });
}
export const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
export function arg(name: string): string | undefined {
const i = process.argv.indexOf(`--${name}`);
return i >= 0 ? process.argv[i + 1] : undefined;
}

View File

@@ -0,0 +1,69 @@
// Import a Spotify data export into music.plays (source 'spotify', origin 'export').
//
// bun scripts/music-import-history.ts <file-or-dir> [...] [--min-ms 30000] [--dry-run]
//
// Accepts the Extended Streaming History (Streaming_History_Audio_*.json, older
// endsong_*.json) and the short account-data export (StreamingHistory_music_*.json).
// Podcasts/audiobooks and plays under --min-ms are skipped. Safe to re-run: a play already
// stored (same artist+title within a minute, also from the live poller) is not added twice.
import { readdirSync, readFileSync, statSync } from 'node:fs';
import { join } from 'node:path';
import { DEFAULT_MIN_MS, parseHistoryEntry, type HistoryPlay } from '../src/lib/music/history';
import { insertPlay, newCache, upsertTrack } from '../src/lib/music/store.server';
import { arg, musicDb } from './music-db';
const minMs = Number(arg('min-ms') ?? DEFAULT_MIN_MS);
const dryRun = process.argv.includes('--dry-run');
const skip = new Set(['--min-ms', arg('min-ms')]);
const inputs = process.argv.slice(2).filter((a) => !a.startsWith('--') && !skip.has(a));
if (!inputs.length) throw new Error('Usage: bun scripts/music-import-history.ts <file-or-dir> ...');
const NAME = /^(Streaming_History_Audio_.*|endsong_?\d*|StreamingHistory(_music)?_?\d*)\.json$/i;
function files(path: string): string[] {
if (statSync(path).isDirectory())
return readdirSync(path).flatMap((name) => {
const child = join(path, name);
return statSync(child).isDirectory() ? files(child) : NAME.test(name) ? [child] : [];
});
return [path];
}
const plays: HistoryPlay[] = [];
let entries = 0;
for (const file of inputs.flatMap(files)) {
const data = JSON.parse(readFileSync(file, 'utf8')) as unknown[];
if (!Array.isArray(data)) continue;
entries += data.length;
for (const entry of data) {
const play = parseHistoryEntry(entry, minMs);
if (play) plays.push(play);
}
console.log(`${file}: ${data.length} entries`);
}
plays.sort((a, b) => a.playedAt.getTime() - b.playedAt.getTime());
console.log(`${plays.length} music plays of ${minMs / 1000}s or more out of ${entries} entries.`);
if (dryRun || !plays.length) process.exit(0);
const sql = musicDb();
try {
const cache = newCache();
let stored = 0;
for (const [i, play] of plays.entries()) {
const trackId = await upsertTrack(sql, play.track, false, cache);
if (
await insertPlay(sql, {
trackId,
playedAt: play.playedAt,
source: 'spotify',
origin: 'export',
msPlayed: play.msPlayed,
platform: play.platform
})
)
stored++;
if ((i + 1) % 5000 === 0) console.log(` ${i + 1}/${plays.length} (${stored} new)`);
}
console.log(`Done: ${stored} new plays stored.`);
} finally {
await sql.end();
}

View File

@@ -0,0 +1,132 @@
// Backfill the full Last.fm history of a user into music.plays (source 'lastfm').
//
// LASTFM_API_KEY=... bun scripts/music-import-lastfm.ts [--user stanrunge] [--sleep 300]
// [--max-pages N] [--reset]
//
// Walks user.getRecentTracks newest → oldest in 200-scrobble pages, always asking for page 1
// of a shrinking [from, to] window, so new scrobbles never shift the pages under it.
// Progress is saved after every page in music.import_state ('lastfm:<user>'): stop it any
// time and re-run to resume; once complete, a re-run only fetches scrobbles newer than the
// last completed run. Overlap with Spotify plays is hidden by the music.plays_deduped view.
// Rate limit: one request per --sleep ms (default 300, Last.fm asks for at most 5/s),
// exponential back-off on 429/5xx and Last.fm errors 8, 11, 16 and 29.
import { parseRecentTracks } from '../src/lib/music/lastfm-parse';
import {
getImportState,
insertPlay,
newCache,
setImportState,
upsertTrack
} from '../src/lib/music/store.server';
import { arg, musicDb, sleep } from './music-db';
type State = {
completeUntil: number;
walk: { from: number; to: number; top: number } | null;
imported: number;
seen: number;
};
const apiKey = process.env.LASTFM_API_KEY;
if (!apiKey) throw new Error('Set LASTFM_API_KEY (https://www.last.fm/api/account/create)');
const user = arg('user') ?? 'stanrunge';
const pause = Number(arg('sleep') ?? 300);
const maxPages = Number(arg('max-pages') ?? Infinity);
const key = `lastfm:${user.toLowerCase()}`;
const RETRYABLE = new Set([8, 11, 16, 29]);
async function fetchPage(from: number, to: number) {
const url = `https://ws.audioscrobbler.com/2.0/?${new URLSearchParams({
method: 'user.getrecenttracks',
user,
api_key: apiKey!,
format: 'json',
limit: '200',
page: '1',
from: String(from),
to: String(to)
})}`;
for (let attempt = 0; ; attempt++) {
let retry: boolean;
let message: string;
try {
const response = await fetch(url, {
headers: { 'User-Agent': 'stanrunge.dev music importer (stan@stanrunge.dev)' },
signal: AbortSignal.timeout(30_000)
});
const body = (await response.json().catch(() => null)) as Record<string, unknown> | null;
if (response.ok && body && !('error' in body)) return body;
const code = Number(body?.error);
message = `HTTP ${response.status}${body?.message ? `: ${String(body.message)}` : ''}`;
retry = response.status === 429 || response.status >= 500 || RETRYABLE.has(code);
} catch (e) {
message = e instanceof Error ? e.message : String(e);
retry = true;
}
if (!retry || attempt >= 6) throw new Error(`Last.fm request failed: ${message}`);
const wait = 2 ** attempt * 1000;
console.warn(` ${message}; retrying in ${wait / 1000}s`);
await sleep(wait);
}
}
const sql = musicDb();
try {
let state: State = (process.argv.includes('--reset')
? null
: await getImportState<State>(sql, key)) ?? {
completeUntil: 0,
walk: null,
imported: 0,
seen: 0
};
if (!state.walk) {
const now = Math.floor(Date.now() / 1000);
state = { ...state, walk: { from: state.completeUntil + 1, to: now, top: now } };
}
const cache = newCache();
let pages = 0;
console.log(
`Last.fm ${user}: importing ${new Date(state.walk!.from * 1000).toISOString()} → ${new Date(state.walk!.to * 1000).toISOString()}`
);
while (state.walk && pages < maxPages) {
const walk = state.walk;
const page =
walk.from > walk.to ? null : parseRecentTracks(await fetchPage(walk.from, walk.to));
pages++;
if (!page || page.plays.length === 0) {
state = { ...state, completeUntil: walk.top, walk: null };
await setImportState(sql, key, state);
console.log(`Done: ${state.imported} new plays stored (${state.seen} scrobbles seen).`);
break;
}
let stored = 0;
for (const play of page.plays) {
const trackId = await upsertTrack(sql, play.track, false, cache);
if (
await insertPlay(sql, {
trackId,
playedAt: play.playedAt,
source: 'lastfm',
origin: 'import'
})
)
stored++;
}
const oldest = Math.min(...page.plays.map((p) => p.uts));
state = {
...state,
imported: state.imported + stored,
seen: state.seen + page.plays.length,
walk: { ...walk, to: oldest >= walk.to ? walk.to - 1 : oldest }
};
await setImportState(sql, key, state);
console.log(
` page ${pages}: ${page.plays.length} scrobbles, ${stored} new, back to ${new Date(oldest * 1000).toISOString()} (~${page.total} left)`
);
await sleep(pause);
}
if (state.walk) console.log(`Stopped after ${pages} pages; re-run to resume.`);
} finally {
await sql.end();
}

18
scripts/music-poll.ts Normal file
View File

@@ -0,0 +1,18 @@
// One Spotify poll (now playing + recently played), for the optional host timer in
// deploy/music-collector. The app normally does this itself every minute (see
// src/lib/server/music/poller.ts); the lease in music.spotify_auth stops double polling.
// MUSIC_DB_URL=... SPOTIFY_CLIENT_ID=... SPOTIFY_CLIENT_SECRET=... bun scripts/music-poll.ts
import { pollSpotify } from '../src/lib/music/collector.server';
import { musicDb } from './music-db';
const clientId = process.env.SPOTIFY_CLIENT_ID;
const clientSecret = process.env.SPOTIFY_CLIENT_SECRET;
if (!clientId || !clientSecret) throw new Error('Set SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET');
const sql = musicDb();
try {
const result = await pollSpotify(sql, { clientId, clientSecret });
console.log(JSON.stringify(result));
if (result.status === 'error') process.exitCode = 1;
} finally {
await sql.end();
}

View File

@@ -1,4 +1,5 @@
import type { Handle } from '@sveltejs/kit/hooks';
import type { Handle, ServerInit } from '@sveltejs/kit/hooks';
import { startMusicPoller } from '#lib/server/music/poller.js';
import { building } from '$app/env';
import { auth } from '#lib/server/auth.js';
import { isOwner, isOwnerOnlyPath, isOwnerWritePath } from '#lib/server/owner.js';
@@ -6,6 +7,11 @@ import { error } from '@sveltejs/kit';
const analytics = /<!-- analytics -->[\s\S]*?<!-- \/analytics -->/;
// Background jobs that live in the app process: the Spotify scrobbler (idle until connected).
export const init: ServerInit = () => {
if (!building) startMusicPoller();
};
export const handle: Handle = async ({ event, resolve }) => {
// Match the endpoint path here: the dev proxy may terminate HTTPS before Vite.
// Better Auth validates request origins against its configured public URL.

View File

@@ -0,0 +1,90 @@
<script lang="ts">
import { relativeTime } from '#lib/relative-time.js';
import Music from 'lucide-svelte/icons/music';
type Listening = {
nowPlaying: boolean;
at: string;
track: string;
artists: string;
album: string | null;
image: string | null;
url: string | null;
};
let { listening, href = '/trackers/music' }: { listening: Listening; href?: string } = $props();
let now = $state(Date.now());
$effect(() => {
const timer = setInterval(() => (now = Date.now()), 30_000);
return () => clearInterval(timer);
});
</script>
<a
{href}
class="flex items-center gap-4 rounded-md border border-[var(--site-border)] bg-[var(--site-surface)] p-3 hover:border-[var(--site-text)]"
aria-label="{listening.nowPlaying
? 'Listening to'
: 'Last played'} {listening.track} by {listening.artists}"
>
{#if listening.image}
<img
src={listening.image}
alt=""
width="64"
height="64"
loading="lazy"
class="size-16 shrink-0 rounded object-cover"
/>
{:else}
<span class="flex size-16 shrink-0 items-center justify-center rounded bg-[var(--site-border)]"
><Music size={24} aria-hidden="true" /></span
>
{/if}
<span class="min-w-0">
<span class="flex items-center gap-2 text-sm text-[var(--site-muted)]">
{#if listening.nowPlaying}
<span class="eq" aria-hidden="true"><i></i><i></i><i></i></span> listening now
{:else}
last played {relativeTime(listening.at, now)}
{/if}
</span>
<span class="block truncate font-medium">{listening.track}</span>
<span class="block truncate text-sm text-[var(--site-muted)]">{listening.artists}</span>
</span>
</a>
<style>
.eq {
display: inline-flex;
align-items: flex-end;
gap: 2px;
height: 12px;
}
.eq i {
width: 3px;
background: #1db954;
animation: eq 0.9s ease-in-out infinite;
}
.eq i:nth-child(2) {
animation-delay: -0.3s;
}
.eq i:nth-child(3) {
animation-delay: -0.6s;
}
@keyframes eq {
0%,
100% {
height: 3px;
}
50% {
height: 12px;
}
}
@media (prefers-reduced-motion: reduce) {
.eq i {
animation: none;
height: 8px;
}
}
</style>

View File

@@ -0,0 +1,177 @@
// Spotify poller: one call does what a scrobbler does every minute.
// 1. GET /me/player → music.now_playing (row 1)
// 2. GET /me/player/recently-played?after=<cursor> → music.plays (Spotify lists plays of
// 30 s or more, max 50 per call, so polling every minute never misses one)
// The authorization (refresh token) lives in music.spotify_auth, written by
// /api/spotify/callback. A lease column keeps two app containers (e.g. mid-deploy) or the
// optional host timer (scripts/music-poll.ts) from polling at the same time.
import { newCache, upsertTrack, insertPlay, type Db } from './store.server';
import { parsePlayer, parseRecentlyPlayed } from './spotify-parse';
export type SpotifyCredentials = { clientId: string; clientSecret: string };
type Fetch = typeof fetch;
export const SPOTIFY_SCOPES = [
'user-read-currently-playing',
'user-read-recently-played',
'user-read-playback-state',
'user-top-read'
] as const;
export class SpotifyError extends Error {
constructor(
readonly status: number,
message: string,
readonly code?: string
) {
super(message);
}
}
type TokenResponse = {
access_token: string;
expires_in: number;
refresh_token?: string;
scope?: string;
};
async function token(
fetchFn: Fetch,
creds: SpotifyCredentials,
body: Record<string, string>
): Promise<TokenResponse> {
const response = await fetchFn('https://accounts.spotify.com/api/token', {
method: 'POST',
headers: {
Authorization: `Basic ${btoa(`${creds.clientId}:${creds.clientSecret}`)}`,
'Content-Type': 'application/x-www-form-urlencoded'
},
body: new URLSearchParams(body),
signal: AbortSignal.timeout(15_000)
});
const json = (await response.json().catch(() => ({}))) as Record<string, unknown>;
if (!response.ok || typeof json.access_token !== 'string')
throw new SpotifyError(
response.status,
`Spotify token request failed (${response.status} ${String(json.error ?? '')})`,
typeof json.error === 'string' ? json.error : undefined
);
return json as unknown as TokenResponse;
}
export const exchangeCode = (
fetchFn: Fetch,
creds: SpotifyCredentials,
code: string,
redirectUri: string
) => token(fetchFn, creds, { grant_type: 'authorization_code', code, redirect_uri: redirectUri });
export const refreshAccessToken = (fetchFn: Fetch, creds: SpotifyCredentials, refresh: string) =>
token(fetchFn, creds, { grant_type: 'refresh_token', refresh_token: refresh });
/** GET a Web API path; null for 204 No Content. */
export async function spotifyGet(fetchFn: Fetch, access: string, path: string) {
const response = await fetchFn(`https://api.spotify.com/v1/${path}`, {
headers: { Authorization: `Bearer ${access}` },
signal: AbortSignal.timeout(15_000)
});
if (response.status === 204) return null;
if (!response.ok)
throw new SpotifyError(
response.status,
`Spotify GET /${path.split('?')[0]} → ${response.status}`
);
const text = await response.text();
return text ? (JSON.parse(text) as unknown) : null;
}
export type PollResult =
| { status: 'idle' }
| { status: 'ok'; playing: boolean; newPlays: number }
| { status: 'error'; error: string };
export async function pollSpotify(
sql: Db,
creds: SpotifyCredentials,
fetchFn: Fetch = fetch
): Promise<PollResult> {
const [auth] = await sql`
update music.spotify_auth set lease_until = now() + interval '55 seconds'
where id = 1 and not needs_reauth and (lease_until is null or lease_until < now())
returning refresh_token, access_token, access_expires_at, recent_cursor`;
if (!auth) return { status: 'idle' };
let error: string | null = null;
let result: PollResult;
try {
let access: string | null = auth.access_token;
const expires = auth.access_expires_at ? new Date(auth.access_expires_at).getTime() : 0;
if (!access || expires < Date.now() + 60_000) {
const fresh = await refreshAccessToken(fetchFn, creds, auth.refresh_token);
access = fresh.access_token;
await sql`
update music.spotify_auth set access_token = ${access},
access_expires_at = now() + make_interval(secs => ${fresh.expires_in}),
refresh_token = coalesce(${fresh.refresh_token ?? null}, refresh_token),
scope = coalesce(${fresh.scope ?? null}, scope), updated_at = now()
where id = 1`;
}
const cache = newCache();
const player = parsePlayer(
await spotifyGet(fetchFn, access!, 'me/player?additional_types=episode')
);
const trackId = player?.track ? await upsertTrack(sql, player.track, true, cache) : null;
await sql`
insert into music.now_playing (id, track_id, is_playing, progress_ms, device_name, device_type, context_uri, updated_at)
values (1, ${trackId}, ${!!(player?.isPlaying && trackId)}, ${player?.progressMs ?? null},
${player?.deviceName ?? null}, ${player?.deviceType ?? null}, ${player?.contextUri ?? null}, now())
on conflict (id) do update set
track_id = coalesce(excluded.track_id, music.now_playing.track_id),
is_playing = excluded.is_playing, progress_ms = excluded.progress_ms,
device_name = excluded.device_name, device_type = excluded.device_type,
context_uri = excluded.context_uri, updated_at = now()`;
const cursor: number | null = auth.recent_cursor === null ? null : Number(auth.recent_cursor);
const recent = parseRecentlyPlayed(
await spotifyGet(
fetchFn,
access!,
`me/player/recently-played?limit=50${cursor ? `&after=${cursor}` : ''}`
)
);
let newPlays = 0;
let newest = cursor ?? 0;
for (const play of recent) {
const at = play.playedAt.getTime();
if (cursor && at <= cursor) continue;
const id = await upsertTrack(sql, play.track, true, cache);
if (
await insertPlay(sql, {
trackId: id,
playedAt: play.playedAt,
source: 'spotify',
origin: 'api',
contextType: play.contextType,
contextUri: play.contextUri
})
)
newPlays++;
newest = Math.max(newest, at);
}
if (newest && newest !== cursor)
await sql`update music.spotify_auth set recent_cursor = ${newest} where id = 1`;
result = { status: 'ok', playing: !!player?.isPlaying, newPlays };
} catch (e) {
error = e instanceof Error ? e.message : String(e);
result = { status: 'error', error };
if (e instanceof SpotifyError && e.code === 'invalid_grant')
// The refresh token was revoked: stop polling until /api/spotify/connect is used again.
await sql`update music.spotify_auth set needs_reauth = true where id = 1`;
else if (e instanceof SpotifyError && e.status === 401)
await sql`update music.spotify_auth set access_token = null where id = 1`;
} finally {
await sql`update music.spotify_auth set lease_until = null, last_poll_at = now(),
last_error = ${error} where id = 1`;
}
return result;
}

55
src/lib/music/history.ts Normal file
View File

@@ -0,0 +1,55 @@
// Pure parser for Spotify data exports, used by scripts/music-import-history.ts:
// - Extended Streaming History (Streaming_History_Audio_*.json, older endsong_*.json):
// { ts, ms_played, master_metadata_track_name, master_metadata_album_artist_name,
// master_metadata_album_album_name, spotify_track_uri, platform, ... }
// - Account data (StreamingHistory_music_*.json / StreamingHistory*.json):
// { endTime: 'YYYY-MM-DD HH:mm' (UTC), artistName, trackName, msPlayed }
// Podcast episodes, audiobooks and plays shorter than minMs are skipped (Spotify's own
// recently-played list only counts plays of 30 s or more, so that is the default).
import type { ParsedTrack } from './spotify-parse';
export type HistoryPlay = {
playedAt: Date;
msPlayed: number;
platform: string | null;
track: ParsedTrack;
};
export const DEFAULT_MIN_MS = 30_000;
type Rec = Record<string, unknown>;
const str = (value: unknown) => (typeof value === 'string' && value.trim() ? value.trim() : null);
export function parseHistoryEntry(value: unknown, minMs = DEFAULT_MIN_MS): HistoryPlay | null {
if (!value || typeof value !== 'object') return null;
const e = value as Rec;
const extended = 'ts' in e;
const name = str(extended ? e.master_metadata_track_name : e.trackName);
const artist = str(extended ? e.master_metadata_album_artist_name : e.artistName);
const msPlayed = Number(extended ? e.ms_played : e.msPlayed);
if (!name || !artist || !Number.isFinite(msPlayed) || msPlayed < minMs) return null;
const rawTime = str(extended ? e.ts : e.endTime);
if (!rawTime) return null;
const playedAt = new Date(
extended ? rawTime : `${rawTime.replace(' ', 'T')}${rawTime.length <= 16 ? ':00' : ''}Z`
);
if (Number.isNaN(playedAt.getTime())) return null;
const uri = str(e.spotify_track_uri);
const id = uri?.match(/^spotify:track:([A-Za-z0-9]{22})$/)?.[1] ?? null;
const album = extended ? str(e.master_metadata_album_album_name) : null;
return {
playedAt,
msPlayed: Math.round(msPlayed),
platform: extended ? str(e.platform) : null,
track: {
spotifyId: id,
name,
durationMs: null,
explicit: null,
artists: [{ spotifyId: null, name: artist }],
album: album
? { spotifyId: null, name: album, imageUrl: null, imageUrlSmall: null, releaseDate: null }
: null
}
};
}

View File

@@ -0,0 +1,66 @@
// Pure parser for Last.fm user.getRecentTracks (format=json, extended=0).
import type { ParsedTrack } from './spotify-parse';
export type LastfmPlay = { playedAt: Date; uts: number; track: ParsedTrack };
export type LastfmPage = { plays: LastfmPlay[]; page: number; totalPages: number; total: number };
type Rec = Record<string, unknown>;
const rec = (value: unknown): Rec =>
value && typeof value === 'object' && !Array.isArray(value) ? (value as Rec) : {};
const text = (value: unknown) => {
const raw = typeof value === 'string' ? value : (rec(value)['#text'] ?? rec(value).name);
return typeof raw === 'string' && raw.trim() ? raw.trim() : null;
};
export function parseRecentTracks(value: unknown): LastfmPage {
const body = rec(rec(value).recenttracks);
const attr = rec(body['@attr']);
const raw = body.track;
const list = Array.isArray(raw) ? raw : raw ? [raw] : [];
const plays: LastfmPlay[] = [];
for (const entry of list) {
const t = rec(entry);
if (rec(t['@attr']).nowplaying === 'true') continue; // not scrobbled yet
const uts = Number(rec(t.date).uts);
const name = text(t.name);
const artist = text(t.artist);
if (!name || !artist || !Number.isFinite(uts) || uts <= 0) continue;
const album = text(t.album);
const images = (Array.isArray(t.image) ? t.image : []).map(rec);
const pick = (size: string) => {
const url = images.find((i) => i.size === size)?.['#text'];
// Last.fm's grey star placeholder is not a cover.
return typeof url === 'string' &&
url.startsWith('https://') &&
!url.includes('2a96cbd8b46e442fc41c2b86b821562f')
? url
: null;
};
plays.push({
uts,
playedAt: new Date(uts * 1000),
track: {
spotifyId: null,
name,
durationMs: null,
explicit: null,
artists: [{ spotifyId: null, name: artist }],
album: album
? {
spotifyId: null,
name: album,
imageUrl: pick('extralarge') ?? pick('large'),
imageUrlSmall: pick('large') ?? pick('medium'),
releaseDate: null
}
: null
}
});
}
return {
plays,
page: Number(attr.page) || 1,
totalPages: Number(attr.totalPages) || 0,
total: Number(attr.total) || 0
};
}

167
src/lib/music/music.test.ts Normal file
View File

@@ -0,0 +1,167 @@
import { describe, expect, it } from 'vitest';
import { parseHistoryEntry } from './history';
import { parseRecentTracks } from './lastfm-parse';
import { matchKey } from './normalize';
import { parsePlayer, parseRecentlyPlayed, parseTrack, pickImages } from './spotify-parse';
const ID = '4uLU6hMCjMI75M1A2tKUQC';
const spotifyTrack = {
type: 'track',
id: ID,
name: 'Never Gonna Give You Up',
duration_ms: 213573,
explicit: false,
artists: [{ id: '0gxyHStUsqpMadRV0Di1Qt', name: 'Rick Astley' }],
album: {
id: '6N9PS4QXF1D0OWPk0Sxtb4',
name: 'Whenever You Need Somebody',
release_date: '1987-11-12',
images: [
{ url: 'https://i.scdn.co/image/640', width: 640, height: 640 },
{ url: 'https://i.scdn.co/image/300', width: 300, height: 300 },
{ url: 'https://i.scdn.co/image/64', width: 64, height: 64 }
]
}
};
describe('spotify parsing', () => {
it('parses a track with album art', () => {
const t = parseTrack(spotifyTrack)!;
expect(t.spotifyId).toBe(ID);
expect(t.artists).toEqual([{ spotifyId: '0gxyHStUsqpMadRV0Di1Qt', name: 'Rick Astley' }]);
expect(t.album?.imageUrl).toBe('https://i.scdn.co/image/640');
expect(t.album?.imageUrlSmall).toBe('https://i.scdn.co/image/300');
});
it('skips local files, episodes and nameless items', () => {
expect(parseTrack({ ...spotifyTrack, is_local: true })).toBeNull();
expect(parseTrack({ ...spotifyTrack, type: 'episode' })).toBeNull();
expect(parseTrack({ ...spotifyTrack, name: ' ' })).toBeNull();
});
it('keeps only https images', () => {
expect(pickImages([{ url: 'http://x/1', width: 640 }])).toEqual({ large: null, small: null });
});
it('sorts recently played oldest first', () => {
const plays = parseRecentlyPlayed({
items: [
{
track: spotifyTrack,
played_at: '2026-10-24T15:00:00.123Z',
context: { type: 'playlist', uri: 'spotify:playlist:x' }
},
{ track: spotifyTrack, played_at: '2026-10-24T14:00:00.000Z', context: null },
{ track: null, played_at: '2026-10-24T13:00:00.000Z' }
]
});
expect(plays.map((p) => p.playedAt.toISOString())).toEqual([
'2026-10-24T14:00:00.000Z',
'2026-10-24T15:00:00.123Z'
]);
expect(plays[1].contextType).toBe('playlist');
});
it('parses the player and ignores podcasts', () => {
expect(parsePlayer(null)).toBeNull();
const p = parsePlayer({
is_playing: true,
progress_ms: 1000,
currently_playing_type: 'track',
item: spotifyTrack,
device: { name: 'Mac', type: 'Computer' }
})!;
expect(p).toMatchObject({ isPlaying: true, deviceName: 'Mac' });
expect(p.track?.name).toBe('Never Gonna Give You Up');
expect(
parsePlayer({ is_playing: true, currently_playing_type: 'episode', item: {} })!.track
).toBeNull();
});
});
describe('history export parsing', () => {
it('parses extended streaming history and skips short plays and podcasts', () => {
const play = parseHistoryEntry({
ts: '2024-01-02T03:04:05Z',
ms_played: 200000,
platform: 'ios',
master_metadata_track_name: 'Song',
master_metadata_album_artist_name: 'Artist',
master_metadata_album_album_name: 'Album',
spotify_track_uri: `spotify:track:${ID}`
})!;
expect(play.track.spotifyId).toBe(ID);
expect(play.playedAt.toISOString()).toBe('2024-01-02T03:04:05.000Z');
expect(
parseHistoryEntry({
ts: '2024-01-02T03:04:05Z',
ms_played: 5000,
master_metadata_track_name: 'S',
master_metadata_album_artist_name: 'A'
})
).toBeNull();
expect(
parseHistoryEntry({
ts: '2024-01-02T03:04:05Z',
ms_played: 900000,
master_metadata_track_name: null,
episode_name: 'Pod'
})
).toBeNull();
});
it('parses the account-data export (UTC endTime)', () => {
const play = parseHistoryEntry({
endTime: '2023-05-06 07:08',
artistName: 'A',
trackName: 'T',
msPlayed: 60000
})!;
expect(play.playedAt.toISOString()).toBe('2023-05-06T07:08:00.000Z');
expect(play.track.spotifyId).toBeNull();
});
});
describe('last.fm parsing', () => {
it('skips the now-playing item and placeholder covers', () => {
const page = parseRecentTracks({
recenttracks: {
'@attr': { page: '1', totalPages: '3', total: '500' },
track: [
{ '@attr': { nowplaying: 'true' }, name: 'Now', artist: { '#text': 'A' } },
{
name: 'Song',
artist: { '#text': 'Artist' },
album: { '#text': 'Album' },
date: { uts: '1700000000' },
image: [
{ size: 'large', '#text': 'https://lastfm.freetls.fastly.net/i/u/174s/abc.png' },
{
size: 'extralarge',
'#text': 'https://lastfm.freetls.fastly.net/i/u/300x300/abc.png'
}
]
},
{
name: 'Starless',
artist: { '#text': 'B' },
date: { uts: '1690000000' },
image: [{ size: 'large', '#text': 'https://x/2a96cbd8b46e442fc41c2b86b821562f.png' }]
}
]
}
});
expect(page.totalPages).toBe(3);
expect(page.plays).toHaveLength(2);
expect(page.plays[0].playedAt.toISOString()).toBe('2023-11-14T22:13:20.000Z');
expect(page.plays[0].track.album?.imageUrl).toContain('300x300');
expect(page.plays[1].track.album).toBeNull();
});
});
describe('matchKey', () => {
it('ignores remaster/feat/accents/punctuation differences', () => {
const base = matchKey('Beyoncé', 'Crazy in Love');
expect(matchKey('Beyonce', 'Crazy In Love (feat. Jay-Z)')).toBe(base);
expect(matchKey('BEYONCÉ', 'Crazy in Love - 2003 Remaster')).toBe(base);
expect(matchKey('Simon & Garfunkel', 'The Boxer')).toBe(
matchKey('Simon and Garfunkel', 'The Boxer')
);
expect(matchKey('A', 'Song - Part 2')).not.toBe(matchKey('A', 'Song'));
});
});

View File

@@ -0,0 +1,28 @@
// Cross-source matching key: the same play reported by Spotify (live API or the streaming
// history export) and Last.fm often differs in spelling ("Song - 2011 Remaster",
// "Song (feat. X)", "&" vs "and", accents). Both sides reduce to "artist|title".
const VERSION_WORDS = /\b(remaster(ed)?|version|edit|mono|stereo|mix|live|deluxe|bonus|single)\b/;
function clean(value: string) {
let s = value
.normalize('NFKD')
.replace(/[\u0300-\u036f]/g, '')
.toLowerCase();
// "(feat. X)", "[with Y]", trailing " feat. Z"
s = s.replace(/\s*[([](feat\.?|ft\.?|featuring|with)\s[^)\]]*[)\]]/g, '');
s = s.replace(/\s+(feat\.?|ft\.?|featuring)\s.*$/, '');
// "(2011 Remaster)", "[Live]" ... and " - 2011 Remaster", " - Radio Edit"
s = s.replace(/\s*[([]([^)\]]*)[)\]]/g, (whole, inner: string) =>
VERSION_WORDS.test(inner) ? '' : whole
);
s = s.replace(/\s+-\s+(.*)$/, (whole, tail: string) => (VERSION_WORDS.test(tail) ? '' : whole));
return s
.replace(/&/g, ' and ')
.replace(/[^\p{L}\p{N}]+/gu, ' ')
.trim();
}
export function matchKey(artist: string, title: string) {
return `${clean(artist)}|${clean(title)}`;
}

View File

@@ -0,0 +1,122 @@
// Pure parsers for Spotify Web API responses (no I/O, no SvelteKit imports), shared by the
// in-app poller, scripts/music-poll.ts and the tests. Spotify trims fields over time (see the
// February 2026 dev-mode changes), so everything is optional and checked.
export type ParsedArtist = { spotifyId: string | null; name: string };
export type ParsedAlbum = {
spotifyId: string | null;
name: string;
imageUrl: string | null;
imageUrlSmall: string | null;
releaseDate: string | null;
};
export type ParsedTrack = {
spotifyId: string | null;
name: string;
durationMs: number | null;
explicit: boolean | null;
artists: ParsedArtist[];
album: ParsedAlbum | null;
};
export type ParsedRecentPlay = {
track: ParsedTrack;
playedAt: Date;
contextType: string | null;
contextUri: string | null;
};
export type ParsedPlayer = {
isPlaying: boolean;
progressMs: number | null;
deviceName: string | null;
deviceType: string | null;
contextUri: string | null;
/** Null for podcasts, ads and unknown items: only music tracks are stored. */
track: ParsedTrack | null;
};
type Rec = Record<string, unknown>;
const rec = (value: unknown): Rec =>
value && typeof value === 'object' && !Array.isArray(value) ? (value as Rec) : {};
const str = (value: unknown) => (typeof value === 'string' && value.trim() ? value.trim() : null);
const int = (value: unknown) =>
typeof value === 'number' && Number.isFinite(value) ? Math.round(value) : null;
const SPOTIFY_ID = /^[A-Za-z0-9]{22}$/;
export const spotifyId = (value: unknown) => {
const id = str(value);
return id && SPOTIFY_ID.test(id) ? id : null;
};
/** Spotify lists images largest first; keep the largest and the smallest of at least 64 px. */
export function pickImages(images: unknown): { large: string | null; small: string | null } {
const list = (Array.isArray(images) ? images : [])
.map(rec)
.filter((i) => typeof i.url === 'string' && (i.url as string).startsWith('https://'))
.map((i) => ({ url: i.url as string, width: int(i.width) ?? 0 }))
.sort((a, b) => b.width - a.width);
if (!list.length) return { large: null, small: null };
const small = [...list].reverse().find((i) => i.width >= 160) ?? list[list.length - 1];
return { large: list[0].url, small: small.url };
}
export function parseTrack(value: unknown): ParsedTrack | null {
const track = rec(value);
if (track.type !== undefined && track.type !== 'track') return null;
if (track.is_local === true) return null;
const name = str(track.name);
if (!name) return null;
const artists = (Array.isArray(track.artists) ? track.artists : [])
.map(rec)
.map((a) => ({ spotifyId: spotifyId(a.id), name: str(a.name) }))
.filter((a): a is ParsedArtist => !!a.name);
if (!artists.length) return null;
const album = rec(track.album);
const albumName = str(album.name);
const images = pickImages(album.images);
return {
spotifyId: spotifyId(track.id),
name,
durationMs: int(track.duration_ms),
explicit: typeof track.explicit === 'boolean' ? track.explicit : null,
artists,
album: albumName
? {
spotifyId: spotifyId(album.id),
name: albumName,
imageUrl: images.large,
imageUrlSmall: images.small,
releaseDate: str(album.release_date)
}
: null
};
}
/** GET /me/player/recently-played → plays, oldest first. */
export function parseRecentlyPlayed(value: unknown): ParsedRecentPlay[] {
const items = rec(value).items;
const plays: ParsedRecentPlay[] = [];
for (const entry of Array.isArray(items) ? items : []) {
const item = rec(entry);
const track = parseTrack(item.track);
const playedAt = str(item.played_at) ? new Date(item.played_at as string) : null;
if (!track || !playedAt || Number.isNaN(playedAt.getTime())) continue;
const context = rec(item.context);
plays.push({ track, playedAt, contextType: str(context.type), contextUri: str(context.uri) });
}
return plays.sort((a, b) => a.playedAt.getTime() - b.playedAt.getTime());
}
/** GET /me/player (null body = 204, nothing active). */
export function parsePlayer(value: unknown): ParsedPlayer | null {
if (value === null || value === undefined) return null;
const player = rec(value);
const device = rec(player.device);
const type = str(player.currently_playing_type);
return {
isPlaying: player.is_playing === true,
progressMs: int(player.progress_ms),
deviceName: str(device.name),
deviceType: str(device.type),
contextUri: str(rec(player.context).uri),
track: type === null || type === 'track' ? parseTrack(player.item) : null
};
}

View File

@@ -0,0 +1,224 @@
// Writes to the music schema with postgres.js. No SvelteKit imports (relative imports only),
// so the app, scripts/music-*.ts and the image's `bun scripts/...` runs all share it.
import type { Sql } from 'postgres';
import { matchKey } from './normalize';
import type { ParsedAlbum, ParsedArtist, ParsedTrack } from './spotify-parse';
// eslint-disable-next-line @typescript-eslint/no-explicit-any
export type Db = Sql<any>;
export type PlaySource = 'spotify' | 'lastfm' | 'youtube' | 'local';
export type PlayOrigin = 'api' | 'export' | 'import' | 'manual';
/** Per-run cache so a poll or import does not look up the same artist/album/track twice. */
export type StoreCache = Map<string, number>;
export const newCache = (): StoreCache => new Map();
export async function upsertArtist(sql: Db, artist: ParsedArtist, cache = newCache()) {
const cacheKey = `artist:${artist.spotifyId ?? `name:${artist.name.toLowerCase()}`}`;
const cached = cache.get(cacheKey);
if (cached) return cached;
let id: number | undefined;
if (artist.spotifyId) {
const [found] =
await sql`select id, name from music.artists where spotify_id = ${artist.spotifyId}`;
if (found) {
id = found.id;
if (found.name !== artist.name)
await sql`update music.artists set name = ${artist.name} where id = ${found.id}`;
} else {
// Adopt a row created by an import (name only) so its plays merge with live ones.
const [adopted] = await sql`
update music.artists set spotify_id = ${artist.spotifyId}, name = ${artist.name}
where id = (select id from music.artists where spotify_id is null
and lower(name) = lower(${artist.name}) order by id limit 1)
returning id`;
id =
adopted?.id ??
(
await sql`
insert into music.artists (spotify_id, name) values (${artist.spotifyId}, ${artist.name})
on conflict (spotify_id) where spotify_id is not null
do update set name = excluded.name returning id`
)[0].id;
}
} else {
const [found] = await sql`
select id from music.artists where lower(name) = lower(${artist.name})
order by (spotify_id is null), id limit 1`;
id =
found?.id ??
(await sql`insert into music.artists (name) values (${artist.name}) returning id`)[0].id;
}
cache.set(cacheKey, id!);
return id!;
}
export async function upsertAlbum(
sql: Db,
album: ParsedAlbum,
artistId: number | null,
cache = newCache()
) {
const cacheKey = `album:${album.spotifyId ?? `name:${artistId}:${album.name.toLowerCase()}`}`;
const cached = cache.get(cacheKey);
if (cached) return cached;
let row: { id: number; image_url: string | null; image_url_small: string | null } | undefined;
if (album.spotifyId) {
[row] =
await sql`select id, image_url, image_url_small from music.albums where spotify_id = ${album.spotifyId}`;
if (!row)
[row] = await sql`
update music.albums set spotify_id = ${album.spotifyId}, name = ${album.name}
where id = (select id from music.albums where spotify_id is null
and lower(name) = lower(${album.name})
and (artist_id is not distinct from ${artistId} or artist_id is null)
order by id limit 1)
returning id, image_url, image_url_small`;
if (!row)
[row] = await sql`
insert into music.albums (spotify_id, name, artist_id, image_url, image_url_small, release_date)
values (${album.spotifyId}, ${album.name}, ${artistId}, ${album.imageUrl},
${album.imageUrlSmall}, ${album.releaseDate})
on conflict (spotify_id) where spotify_id is not null
do update set name = excluded.name returning id, image_url, image_url_small`;
} else {
[row] = await sql`
select id, image_url, image_url_small from music.albums
where lower(name) = lower(${album.name}) and artist_id is not distinct from ${artistId}
order by (spotify_id is null), id limit 1`;
if (!row)
[row] = await sql`
insert into music.albums (name, artist_id, image_url, image_url_small)
values (${album.name}, ${artistId}, ${album.imageUrl}, ${album.imageUrlSmall})
returning id, image_url, image_url_small`;
}
// Spotify covers win; an import only fills a missing cover.
const better = album.spotifyId
? album.imageUrl && album.imageUrl !== row!.image_url
: album.imageUrl && !row!.image_url;
if (better)
await sql`update music.albums set image_url = ${album.imageUrl},
image_url_small = ${album.imageUrlSmall ?? album.imageUrl},
release_date = coalesce(${album.releaseDate}, release_date) where id = ${row!.id}`;
cache.set(cacheKey, row!.id);
return row!.id;
}
/**
* Find or create a track. `full` = complete Spotify API data (all artists, album, duration):
* it may correct an existing row; partial import data never overwrites anything.
*/
export async function upsertTrack(sql: Db, track: ParsedTrack, full: boolean, cache = newCache()) {
const artistIds: number[] = [];
for (const artist of track.artists) artistIds.push(await upsertArtist(sql, artist, cache));
const artistId = artistIds[0];
const cacheKey = `track:${track.spotifyId ?? `name:${artistId}:${track.name.toLowerCase()}`}`;
const cached = cache.get(cacheKey);
if (cached) return cached;
const albumId = track.album ? await upsertAlbum(sql, track.album, artistId, cache) : null;
const key = matchKey(track.artists[0].name, track.name);
let id: number | undefined;
if (track.spotifyId) {
const [found] = await sql`
select id, album_id, artist_id, duration_ms from music.tracks where spotify_id = ${track.spotifyId}`;
if (found) {
id = found.id;
if (
full &&
(found.album_id !== albumId ||
found.artist_id !== artistId ||
found.duration_ms !== track.durationMs)
)
await sql`update music.tracks set name = ${track.name}, album_id = ${albumId},
artist_id = ${artistId}, duration_ms = ${track.durationMs},
explicit = ${track.explicit}, match_key = ${key} where id = ${id}`;
else if (!found.album_id && albumId)
await sql`update music.tracks set album_id = ${albumId} where id = ${id}`;
} else {
const [adopted] = await sql`
update music.tracks set spotify_id = ${track.spotifyId},
album_id = coalesce(${albumId}, album_id),
duration_ms = coalesce(${track.durationMs}, duration_ms),
explicit = coalesce(${track.explicit}, explicit)
where id = (select id from music.tracks where spotify_id is null
and lower(name) = lower(${track.name}) and artist_id = ${artistId}
order by id limit 1)
returning id`;
id =
adopted?.id ??
(
await sql`
insert into music.tracks (spotify_id, name, album_id, artist_id, duration_ms, explicit, match_key)
values (${track.spotifyId}, ${track.name}, ${albumId}, ${artistId},
${track.durationMs}, ${track.explicit}, ${key})
on conflict (spotify_id) where spotify_id is not null
do update set name = excluded.name returning id`
)[0].id;
}
} else {
const [found] = await sql`
select id, album_id from music.tracks
where lower(name) = lower(${track.name}) and artist_id = ${artistId}
order by (spotify_id is null), id limit 1`;
if (found) {
id = found.id;
if (!found.album_id && albumId)
await sql`update music.tracks set album_id = ${albumId} where id = ${id}`;
} else
id = (
await sql`
insert into music.tracks (name, album_id, artist_id, duration_ms, match_key)
values (${track.name}, ${albumId}, ${artistId}, ${track.durationMs}, ${key})
returning id`
)[0].id;
}
for (const [position, aid] of artistIds.entries())
await sql`insert into music.track_artists (track_id, artist_id, position)
values (${id!}, ${aid}, ${position}) on conflict do nothing`;
cache.set(cacheKey, id!);
return id!;
}
export type PlayInput = {
trackId: number;
playedAt: Date;
source: PlaySource;
origin: PlayOrigin;
msPlayed?: number | null;
contextType?: string | null;
contextUri?: string | null;
platform?: string | null;
};
/**
* Insert a play unless the same source already has it: same normalised artist+title within
* a minute (the live API and the Spotify export report the same end time to within seconds;
* Last.fm re-imports have the exact same timestamp). Cross-source overlap (Last.fm vs
* Spotify) is resolved at read time by the music.plays_deduped view. Returns true if stored.
*/
export async function insertPlay(sql: Db, play: PlayInput) {
const at = play.playedAt.toISOString();
const rows = await sql`
insert into music.plays (track_id, played_at, source, origin, ms_played, context_type, context_uri, platform)
select ${play.trackId}, ${at}::timestamptz, ${play.source}, ${play.origin}, ${play.msPlayed ?? null},
${play.contextType ?? null}, ${play.contextUri ?? null}, ${play.platform ?? null}
where not exists (
select 1 from music.plays p join music.tracks t on t.id = p.track_id
where p.source = ${play.source}
and p.played_at between ${at}::timestamptz - interval '60 seconds'
and ${at}::timestamptz + interval '60 seconds'
and t.match_key = (select match_key from music.tracks where id = ${play.trackId}))
on conflict do nothing
returning id`;
return rows.length > 0;
}
export async function getImportState<T>(sql: Db, key: string): Promise<T | null> {
const [row] = await sql`select data from music.import_state where key = ${key}`;
return (row?.data as T) ?? null;
}
export async function setImportState(sql: Db, key: string, data: unknown) {
await sql`insert into music.import_state (key, data) values (${key}, ${sql.json(data as never)})
on conflict (key) do update set data = excluded.data, updated_at = now()`;
}

View File

@@ -65,6 +65,13 @@ const allTrackers = [
external: false,
private: false
},
{
name: 'music',
subtitle: 'everything I listen to: top artists, tracks, albums and streaks',
href: '/trackers/music',
external: false,
private: false
},
{
name: 'health',
subtitle: 'private: body, workouts, sleep, meds and vaccinations',

View File

@@ -3,7 +3,8 @@ import { relations } from './relations';
import { drizzle } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
const client = postgres(DB_URL, { prepare: false });
/** Raw postgres.js client (same pool), for hand-written SQL such as the music stats. */
export const client = postgres(DB_URL, { prepare: false });
export const db = drizzle({
relations,

View File

@@ -18,3 +18,4 @@ export * from './tasks';
export * from './groceries';
export * from './home';
export * from './cars';
export * from './music';

View File

@@ -0,0 +1,180 @@
import { sql } from 'drizzle-orm';
import {
bigint,
boolean,
check,
index,
integer,
jsonb,
primaryKey,
smallint,
text,
uniqueIndex
} from 'drizzle-orm/pg-core';
import { musicSchema } from './schemas';
import { id, timestamptz } from './helpers';
// Listening history (a self-hosted Last.fm): every play from Spotify (polled every minute,
// see src/lib/server/music/collector.ts) and later YouTube / local files, plus the
// Extended Streaming History import (scripts/music-import-history.ts). Public read
// (personal_ro gets SELECT) except music.spotify_auth, which holds the refresh token.
// Rows without a spotify_id (imports, YouTube, local files) are matched by name.
export const musicArtists = musicSchema.table(
'artists',
{
id: id(),
spotifyId: text('spotify_id'),
name: text('name').notNull(),
imageUrl: text('image_url'),
createdAt: timestamptz('created_at').notNull().defaultNow()
},
(t) => [
uniqueIndex('artists_spotify_id_key')
.on(t.spotifyId)
.where(sql`${t.spotifyId} is not null`),
index('artists_name_idx').on(sql`lower(${t.name})`)
]
);
export const musicAlbums = musicSchema.table(
'albums',
{
id: id(),
spotifyId: text('spotify_id'),
name: text('name').notNull(),
artistId: integer('artist_id').references(() => musicArtists.id, { onDelete: 'set null' }),
/** Largest cover (640 px on Spotify) and a small one (64–300 px) for lists. */
imageUrl: text('image_url'),
imageUrlSmall: text('image_url_small'),
releaseDate: text('release_date'),
createdAt: timestamptz('created_at').notNull().defaultNow()
},
(t) => [
uniqueIndex('albums_spotify_id_key')
.on(t.spotifyId)
.where(sql`${t.spotifyId} is not null`),
index('albums_name_idx').on(sql`lower(${t.name})`)
]
);
export const musicTracks = musicSchema.table(
'tracks',
{
id: id(),
spotifyId: text('spotify_id'),
name: text('name').notNull(),
albumId: integer('album_id').references(() => musicAlbums.id, { onDelete: 'set null' }),
/** Primary (first credited) artist, also in track_artists at position 0. */
artistId: integer('artist_id').references(() => musicArtists.id, { onDelete: 'set null' }),
durationMs: integer('duration_ms'),
explicit: boolean('explicit'),
/** normalised "primary artist|title" (src/lib/music/normalize.ts), for cross-source dedupe. */
matchKey: text('match_key').notNull(),
createdAt: timestamptz('created_at').notNull().defaultNow()
},
(t) => [
uniqueIndex('tracks_spotify_id_key')
.on(t.spotifyId)
.where(sql`${t.spotifyId} is not null`),
index('tracks_name_idx').on(sql`lower(${t.name})`),
index('tracks_match_key_idx').on(t.matchKey)
]
);
export const musicTrackArtists = musicSchema.table(
'track_artists',
{
trackId: integer('track_id')
.notNull()
.references(() => musicTracks.id, { onDelete: 'cascade' }),
artistId: integer('artist_id')
.notNull()
.references(() => musicArtists.id, { onDelete: 'cascade' }),
position: smallint('position').notNull().default(0)
},
(t) => [
primaryKey({ columns: [t.trackId, t.artistId] }),
index('track_artists_artist_idx').on(t.artistId)
]
);
export const musicPlays = musicSchema.table(
'plays',
{
id: id(),
trackId: integer('track_id')
.notNull()
.references(() => musicTracks.id, { onDelete: 'cascade' }),
/** Spotify: when the play ended (played_at / export ts). Last.fm: the scrobble time,
* usually when the track started. See the music.plays_deduped view. */
playedAt: timestamptz('played_at').notNull(),
source: text('source').notNull().default('spotify'),
/** How the row got here: 'api' (poller), 'export' (Spotify streaming history),
* 'import' (Last.fm and other backfills), 'manual'. */
origin: text('origin').notNull().default('api'),
/** Known for imports; null for API plays (Spotify only reports plays of 30 s or more). */
msPlayed: integer('ms_played'),
contextType: text('context_type'),
contextUri: text('context_uri'),
platform: text('platform'),
createdAt: timestamptz('created_at').notNull().defaultNow()
},
(t) => [
uniqueIndex('plays_dedupe_key').on(t.source, t.trackId, t.playedAt),
index('plays_played_at_idx').on(t.playedAt),
index('plays_source_played_idx').on(t.source, t.playedAt),
index('plays_track_played_idx').on(t.trackId, t.playedAt),
check('plays_source_check', sql`${t.source} in ('spotify', 'lastfm', 'youtube', 'local')`),
check('plays_origin_check', sql`${t.origin} in ('api', 'export', 'import', 'manual')`)
]
);
/** One row (id = 1): what is playing right now, refreshed by the poller every minute. */
export const musicNowPlaying = musicSchema.table(
'now_playing',
{
id: smallint('id').primaryKey().default(1),
trackId: integer('track_id').references(() => musicTracks.id, { onDelete: 'set null' }),
isPlaying: boolean('is_playing').notNull().default(false),
progressMs: integer('progress_ms'),
deviceName: text('device_name'),
deviceType: text('device_type'),
contextUri: text('context_uri'),
updatedAt: timestamptz('updated_at').notNull().defaultNow()
},
(t) => [check('now_playing_single_row', sql`${t.id} = 1`)]
);
/**
* One row (id = 1): the owner's Spotify authorization (Authorization Code flow, stored by
* /api/spotify/callback) and the poller's state. Never granted to personal_ro.
*/
export const musicSpotifyAuth = musicSchema.table(
'spotify_auth',
{
id: smallint('id').primaryKey().default(1),
spotifyUserId: text('spotify_user_id').notNull(),
displayName: text('display_name'),
refreshToken: text('refresh_token').notNull(),
accessToken: text('access_token'),
accessExpiresAt: timestamptz('access_expires_at'),
scope: text('scope'),
/** recently-played `after` cursor: played_at (ms since epoch) of the newest stored play. */
recentCursor: bigint('recent_cursor', { mode: 'number' }),
needsReauth: boolean('needs_reauth').notNull().default(false),
leaseUntil: timestamptz('lease_until'),
lastPollAt: timestamptz('last_poll_at'),
lastError: text('last_error'),
connectedAt: timestamptz('connected_at').notNull().defaultNow(),
updatedAt: timestamptz('updated_at').notNull().defaultNow()
},
(t) => [check('spotify_auth_single_row', sql`${t.id} = 1`)]
);
/** Resumable backfill state, e.g. key 'lastfm:stanrunge' (see scripts/music-import-lastfm.ts). */
export const musicImportState = musicSchema.table('import_state', {
key: text('key').primaryKey(),
data: jsonb('data').notNull(),
updatedAt: timestamptz('updated_at').notNull().defaultNow()
});

View File

@@ -18,3 +18,4 @@ export const tasksSchema = pgSchema('tasks');
export const groceriesSchema = pgSchema('groceries');
export const homeSchema = pgSchema('home');
export const carsSchema = pgSchema('cars');
export const musicSchema = pgSchema('music');

View File

@@ -0,0 +1,33 @@
import { SPOTIFY_SCOPES } from '#lib/music/collector.server.js';
export const STATE_COOKIE = 'spotify_oauth_state';
export const CALLBACK_PATH = '/api/spotify/callback';
/**
* The redirect URI registered in the Spotify app: https://stanrunge.dev/api/spotify/callback
* (prod) or https://dev.stanrunge.dev/api/spotify/callback (dev server). Caddy terminates TLS,
* so the request URL may say http; the public hosts are always https.
*/
export function publicOrigin(url: URL) {
if (url.hostname === 'stanrunge.dev' || url.hostname.endsWith('.stanrunge.dev'))
return `https://${url.hostname}`;
return url.origin;
}
export function authorizeUrl(clientId: string, redirectUri: string, state: string) {
return `https://accounts.spotify.com/authorize?${new URLSearchParams({
response_type: 'code',
client_id: clientId,
scope: SPOTIFY_SCOPES.join(' '),
redirect_uri: redirectUri,
state,
show_dialog: 'true'
})}`;
}
export function safeEqual(a: string, b: string) {
if (a.length !== b.length) return false;
let diff = 0;
for (let i = 0; i < a.length; i++) diff |= a.charCodeAt(i) ^ b.charCodeAt(i);
return diff === 0;
}

View File

@@ -0,0 +1,43 @@
import { SPOTIFY_CLIENT_ID, SPOTIFY_CLIENT_SECRET } from '$app/env/private';
import { client } from '#lib/server/db/index.js';
import { pollSpotify } from '#lib/music/collector.server.js';
// The Spotify poller runs inside the app (started from hooks.server.ts `init`): the app
// already has the database role (personal_app) and the Spotify client credentials, so
// no extra role, secret file or host timer is needed. It does nothing until the owner has
// connected Spotify at /api/spotify/connect. MUSIC_POLLER=off disables it (e.g. when the
// optional host timer, deploy/music-collector, is used instead).
const INTERVAL_MS = 60_000;
let timer: ReturnType<typeof setInterval> | undefined;
let running = false;
let lastLogged = '';
export const spotifyCredentials = () => ({
clientId: SPOTIFY_CLIENT_ID,
clientSecret: SPOTIFY_CLIENT_SECRET
});
export async function pollNow() {
if (running || !SPOTIFY_CLIENT_ID || !SPOTIFY_CLIENT_SECRET) return;
running = true;
try {
const result = await pollSpotify(client, spotifyCredentials());
const message = result.status === 'error' ? result.error : '';
if (message && message !== lastLogged) console.error(`[music] poll failed: ${message}`);
lastLogged = message;
} catch (e) {
// e.g. the music schema is not migrated on this database yet; log once, keep trying.
const message = e instanceof Error ? e.message : String(e);
if (message !== lastLogged) console.error(`[music] poll failed: ${message}`);
lastLogged = message;
} finally {
running = false;
}
}
export function startMusicPoller() {
if (timer || process.env.MUSIC_POLLER === 'off') return;
if (!SPOTIFY_CLIENT_ID || !SPOTIFY_CLIENT_SECRET) return;
timer = setInterval(pollNow, INTERVAL_MS);
setTimeout(pollNow, 5_000);
}

View File

@@ -0,0 +1,212 @@
import { client as sql } from '#lib/server/db/index.js';
// Read side of the listening tracker. Everything reads music.plays_deduped, which hides a
// Last.fm scrobble when Spotify (live API or export) has the same play, so nothing is
// counted twice. Listening time uses ms_played when known (imports) and otherwise the
// track length (the API only reports plays of 30 s or more, not how long).
const TZ = 'Europe/Amsterdam';
// Fragments are built per query (postgres.js queries are single-use objects).
const playMs = () => sql`coalesce(p.ms_played, t.duration_ms, 210000)`;
export const RANGES = { '7d': 7, '30d': 30, all: null } as const;
export type Range = keyof typeof RANGES;
export const parseRange = (value: string | null): Range =>
value && value in RANGES ? (value as Range) : '30d';
export type Listening = {
nowPlaying: boolean;
at: string;
track: string;
artists: string;
album: string | null;
image: string | null;
url: string | null;
device: string | null;
};
const artistsOf = () => sql`coalesce((select string_agg(ar.name, ', ' order by ta.position)
from music.track_artists ta join music.artists ar on ar.id = ta.artist_id
where ta.track_id = t.id), a.name, '')`;
const toListening = (row: Record<string, unknown>, nowPlaying: boolean): Listening => ({
nowPlaying,
at: new Date(row.at as string).toISOString(),
track: row.track as string,
artists: row.artists as string,
album: (row.album as string) ?? null,
image: (row.image as string) ?? null,
url: row.spotify_id ? `https://open.spotify.com/track/${row.spotify_id}` : null,
device: (row.device as string) ?? null
});
/** Homepage card: what is playing now (fresh within 3 min), else the last play. Never throws. */
export async function getListening(): Promise<Listening | null> {
try {
const [now] = await sql`
select np.updated_at as at, t.name as track, t.spotify_id, ${artistsOf()} as artists,
al.name as album, coalesce(al.image_url_small, al.image_url) as image, np.device_name as device
from music.now_playing np
join music.tracks t on t.id = np.track_id
left join music.artists a on a.id = t.artist_id
left join music.albums al on al.id = t.album_id
where np.id = 1 and np.is_playing and np.updated_at > now() - interval '3 minutes'`;
if (now) return toListening(now, true);
const [last] = await sql`
select p.played_at as at, t.name as track, t.spotify_id, ${artistsOf()} as artists,
al.name as album, coalesce(al.image_url_small, al.image_url) as image, null as device
from music.plays p
join music.tracks t on t.id = p.track_id
left join music.artists a on a.id = t.artist_id
left join music.albums al on al.id = t.album_id
order by p.played_at desc limit 1`;
return last ? toListening(last, false) : null;
} catch {
return null; // e.g. not migrated on this database: the homepage must still render
}
}
export async function getMusicStats(range: Range) {
const days = RANGES[range];
const since = (days ? new Date(Date.now() - days * 86_400_000) : new Date(0)).toISOString();
const chartDays = days ?? 90;
const [totals, artists, tracks, albums, daily, streak, recent, sources] = await Promise.all([
sql`
select count(*)::int as plays,
coalesce(sum(${playMs()}), 0)::float8 / 3600000 as hours,
count(distinct p.track_id)::int as tracks,
count(distinct t.artist_id)::int as artists,
min(p.played_at) as first
from music.plays_deduped p join music.tracks t on t.id = p.track_id
where p.played_at >= ${since}::timestamptz`,
sql`
select a.id, a.name, a.spotify_id, count(*)::int as plays,
coalesce(sum(${playMs()}), 0)::float8 / 3600000 as hours,
coalesce(a.image_url, (select coalesce(al.image_url_small, al.image_url) from music.albums al
where al.artist_id = a.id and al.image_url is not null
order by al.spotify_id is null, al.id desc limit 1)) as image
from music.plays_deduped p
join music.tracks t on t.id = p.track_id
join music.artists a on a.id = t.artist_id
where p.played_at >= ${since}::timestamptz
group by a.id order by plays desc, a.name limit 10`,
sql`
select t.id, t.name, t.spotify_id, ${artistsOf()} as artists, count(*)::int as plays,
coalesce(al.image_url_small, al.image_url) as image
from music.plays_deduped p
join music.tracks t on t.id = p.track_id
left join music.artists a on a.id = t.artist_id
left join music.albums al on al.id = t.album_id
where p.played_at >= ${since}::timestamptz
group by t.id, a.name, al.id order by plays desc, t.name limit 10`,
sql`
select al.id, al.name, al.spotify_id, a.name as artist, count(*)::int as plays,
coalesce(al.image_url_small, al.image_url) as image
from music.plays_deduped p
join music.tracks t on t.id = p.track_id
join music.albums al on al.id = t.album_id
left join music.artists a on a.id = al.artist_id
where p.played_at >= ${since}::timestamptz
group by al.id, a.name order by plays desc, al.name limit 10`,
sql`
with days as (
select generate_series((now() at time zone ${TZ})::date - ${chartDays - 1}::int,
(now() at time zone ${TZ})::date, interval '1 day')::date as day)
select d.day::text as day, count(p.id)::int as plays,
coalesce(sum(${playMs()}), 0)::float8 / 3600000 as hours
from days d
left join music.plays_deduped p
on p.played_at >= now() - make_interval(days => ${chartDays + 1})
and (p.played_at at time zone ${TZ})::date = d.day
left join music.tracks t on t.id = p.track_id
group by d.day order by d.day`,
sql`
with d as (select distinct (played_at at time zone ${TZ})::date as day from music.plays_deduped),
g as (select day, day - (row_number() over (order by day))::int as grp from d),
s as (select min(day) as start_day, max(day) as end_day, count(*)::int as len from g group by grp)
select
coalesce((select len from s where end_day >= (now() at time zone ${TZ})::date - 1
order by end_day desc limit 1), 0) as current,
coalesce((select len from s order by len desc, end_day desc limit 1), 0) as longest,
(select start_day::text from s order by len desc, end_day desc limit 1) as longest_start,
(select end_day::text from s order by len desc, end_day desc limit 1) as longest_end,
(select count(*)::int from d) as days`,
sql`
select p.id, p.played_at, p.source, t.name as track, t.spotify_id, ${artistsOf()} as artists,
coalesce(al.image_url_small, al.image_url) as image
from music.plays_deduped p
join music.tracks t on t.id = p.track_id
left join music.artists a on a.id = t.artist_id
left join music.albums al on al.id = t.album_id
order by p.played_at desc limit 50`,
sql`select source, count(*)::int as plays from music.plays_deduped group by source order by plays desc`
]);
const t = totals[0];
return {
range,
totals: {
plays: t.plays as number,
hours: Number(t.hours),
tracks: t.tracks as number,
artists: t.artists as number,
first: t.first ? new Date(t.first).toISOString() : null
},
topArtists: artists.map((r) => ({
name: r.name as string,
plays: r.plays as number,
hours: Number(r.hours),
image: (r.image as string) ?? null,
url: r.spotify_id ? `https://open.spotify.com/artist/${r.spotify_id}` : null
})),
topTracks: tracks.map((r) => ({
name: r.name as string,
artists: r.artists as string,
plays: r.plays as number,
image: (r.image as string) ?? null,
url: r.spotify_id ? `https://open.spotify.com/track/${r.spotify_id}` : null
})),
topAlbums: albums.map((r) => ({
name: r.name as string,
artist: (r.artist as string) ?? '',
plays: r.plays as number,
image: (r.image as string) ?? null,
url: r.spotify_id ? `https://open.spotify.com/album/${r.spotify_id}` : null
})),
daily: daily.map((r) => ({
day: r.day as string,
plays: r.plays as number,
hours: Number(r.hours)
})),
streak: {
current: Number(streak[0].current),
longest: Number(streak[0].longest),
longestStart: (streak[0].longest_start as string) ?? null,
longestEnd: (streak[0].longest_end as string) ?? null,
days: Number(streak[0].days)
},
recent: recent.map((r) => ({
id: r.id as number,
at: new Date(r.played_at).toISOString(),
source: r.source as string,
track: r.track as string,
artists: r.artists as string,
image: (r.image as string) ?? null,
url: r.spotify_id ? `https://open.spotify.com/track/${r.spotify_id}` : null
})),
sources: sources.map((r) => ({ source: r.source as string, plays: r.plays as number }))
};
}
/** Owner-only connection status for /trackers/music. Never includes tokens. */
export async function getSpotifyConnection() {
const [row] = await sql`
select spotify_user_id, display_name, needs_reauth, last_poll_at, last_error, connected_at
from music.spotify_auth where id = 1`;
if (!row) return null;
return {
user: (row.display_name as string) || (row.spotify_user_id as string),
needsReauth: row.needs_reauth as boolean,
lastPollAt: row.last_poll_at ? new Date(row.last_poll_at).toISOString() : null,
lastError: (row.last_error as string) ?? null,
connectedAt: new Date(row.connected_at).toISOString()
};
}

View File

@@ -26,7 +26,9 @@ export const ownerOnlyPrefixes = [
'/trackers/groceries',
'/trackers/houses',
'/trackers/books/files',
'/lists/db'
'/lists/db',
// Spotify OAuth for the listening tracker (connect + callback 404 for everyone else).
'/api/spotify'
] as const;
/**
@@ -46,7 +48,8 @@ export const ownerWritePrefixes = [
'/trackers/groceries',
'/trackers/houses',
'/trackers/books',
'/trackers/cars'
'/trackers/cars',
'/trackers/music'
] as const;
const normalise = (pathname: string) => pathname.toLowerCase().replace(/\/+$/, '');

View File

@@ -1,10 +1,12 @@
import { fail } from '@sveltejs/kit';
import { listSummaries } from '#lib/server/lists.js';
import { getListening } from '#lib/server/music/stats.js';
import type { PageServerLoad } from './$types';
export const load: PageServerLoad = async () => {
// `done` is a property of the item now (one owner), so progress is the same for every visitor.
return { lists: await listSummaries() };
const [lists, listening] = await Promise.all([listSummaries(), getListening()]);
return { lists, listening };
};
// Retire the old shared-token login; legacy cookies grant no permissions.

View File

@@ -2,6 +2,7 @@
import { resolve } from '$app/paths';
import portrait from '#lib/assets/stan.jpg?url';
import ProgressBar from '#lib/components/ProgressBar.svelte';
import ListeningCard from '#lib/music/ListeningCard.svelte';
import CatalogGrid from '#lib/components/CatalogGrid.svelte';
import { projects, visibleTrackers } from '#lib/projects.js';
import ArrowRight from 'lucide-svelte/icons/arrow-right';
@@ -99,6 +100,11 @@
>email <Mail size={14} aria-hidden="true" /></a
>
</nav>
{#if data.listening}
<div class="mx-auto mt-7 max-w-sm text-left md:mx-0">
<ListeningCard listening={data.listening} />
</div>
{/if}
</div>
<img
src={portrait}

View File

@@ -0,0 +1,47 @@
import { error, redirect } from '@sveltejs/kit';
import { client } from '#lib/server/db/index.js';
import { requireOwner } from '#lib/server/owner.js';
import { exchangeCode, spotifyGet } from '#lib/music/collector.server.js';
import { CALLBACK_PATH, publicOrigin, safeEqual, STATE_COOKIE } from '#lib/server/music/oauth.js';
import { pollNow, spotifyCredentials } from '#lib/server/music/poller.js';
import type { RequestHandler } from './$types';
// Spotify redirects the owner here after /api/spotify/connect. Stores the refresh token in
// music.spotify_auth (app role only) and kicks off the first poll.
export const GET: RequestHandler = async (event) => {
requireOwner(event);
const { url, cookies, fetch } = event;
const expected = cookies.get(STATE_COOKIE) ?? '';
cookies.delete(STATE_COOKIE, { path: '/api/spotify' });
const state = url.searchParams.get('state') ?? '';
if (!expected || !safeEqual(state, expected)) error(400, 'Invalid or expired state, try again.');
if (url.searchParams.get('error'))
redirect(303, `/trackers/music?spotify=${encodeURIComponent(url.searchParams.get('error')!)}`);
const code = url.searchParams.get('code');
if (!code) error(400, 'Missing code');
const creds = spotifyCredentials();
if (!creds.clientId || !creds.clientSecret)
error(503, 'Spotify is not configured on this server.');
const tokens = await exchangeCode(fetch, creds, code, `${publicOrigin(url)}${CALLBACK_PATH}`);
if (!tokens.refresh_token) error(502, 'Spotify did not return a refresh token.');
const me = (await spotifyGet(fetch, tokens.access_token, 'me')) as Record<string, unknown> | null;
const userId = typeof me?.id === 'string' ? me.id : 'unknown';
const name = typeof me?.display_name === 'string' ? me.display_name : null;
await client`
insert into music.spotify_auth (id, spotify_user_id, display_name, refresh_token, access_token,
access_expires_at, scope, needs_reauth, last_error, connected_at, updated_at)
values (1, ${userId}, ${name}, ${tokens.refresh_token}, ${tokens.access_token},
now() + make_interval(secs => ${tokens.expires_in}), ${tokens.scope ?? null}, false, null, now(), now())
on conflict (id) do update set
spotify_user_id = excluded.spotify_user_id, display_name = excluded.display_name,
refresh_token = excluded.refresh_token, access_token = excluded.access_token,
access_expires_at = excluded.access_expires_at, scope = excluded.scope,
needs_reauth = false, last_error = null, lease_until = null,
-- another account starts a fresh recently-played cursor
recent_cursor = case when music.spotify_auth.spotify_user_id = excluded.spotify_user_id
then music.spotify_auth.recent_cursor end,
connected_at = now(), updated_at = now()`;
void pollNow();
redirect(303, '/trackers/music?spotify=connected');
};

View File

@@ -0,0 +1,29 @@
import { SPOTIFY_CLIENT_ID } from '$app/env/private';
import { error, redirect } from '@sveltejs/kit';
import { requireOwner } from '#lib/server/owner.js';
import {
authorizeUrl,
CALLBACK_PATH,
publicOrigin,
STATE_COOKIE
} from '#lib/server/music/oauth.js';
import type { RequestHandler } from './$types';
// Owner only: start the Spotify Authorization Code flow for the listening tracker.
export const GET: RequestHandler = (event) => {
requireOwner(event);
const { url, cookies } = event;
// The state cookie must be set on the host Spotify redirects back to.
if (url.hostname === 'www.stanrunge.dev')
redirect(302, 'https://stanrunge.dev/api/spotify/connect');
if (!SPOTIFY_CLIENT_ID) error(503, 'Spotify is not configured on this server.');
const state = Buffer.from(crypto.getRandomValues(new Uint8Array(32))).toString('base64url');
cookies.set(STATE_COOKIE, state, {
path: '/api/spotify',
httpOnly: true,
secure: true,
sameSite: 'lax',
maxAge: 600
});
redirect(302, authorizeUrl(SPOTIFY_CLIENT_ID, `${publicOrigin(url)}${CALLBACK_PATH}`, state));
};

View File

@@ -15,6 +15,7 @@ const staticPaths = [
'/lists',
'/trackers/books',
'/trackers/cars',
'/trackers/music',
'/projects/game-tracker',
'/projects/game-tracker/pokemon',
'/projects/game-tracker/osu',

View File

@@ -0,0 +1,28 @@
import { isOwner } from '#lib/server/owner.js';
import {
getListening,
getMusicStats,
getSpotifyConnection,
parseRange
} from '#lib/server/music/stats.js';
import type { PageServerLoad } from './$types';
export const load: PageServerLoad = async (event) => {
const range = parseRange(event.url.searchParams.get('range'));
const owner = isOwner(event);
const [stats, listening, connection] = await Promise.all([
getMusicStats(range),
getListening(),
owner ? getSpotifyConnection() : Promise.resolve(null)
]);
return {
stats,
listening,
connection,
spotifyStatus: owner ? event.url.searchParams.get('spotify') : null,
seo: {
title: 'Music',
description: 'What I listen to: top artists, tracks and albums, listening time and streaks.'
}
};
};

View File

@@ -0,0 +1,251 @@
<script lang="ts">
import ListeningCard from '#lib/music/ListeningCard.svelte';
import { relativeTime } from '#lib/relative-time.js';
import Music from 'lucide-svelte/icons/music';
import type { PageProps } from './$types';
let { data }: PageProps = $props();
const s = $derived(data.stats);
const ranges = [
['7d', '7 days'],
['30d', '30 days'],
['all', 'all time']
] as const;
const maxHours = $derived(Math.max(0.01, ...s.daily.map((d) => d.hours)));
const fmtHours = (h: number) =>
h >= 10
? Math.round(h).toLocaleString('en-GB')
: h.toLocaleString('en-GB', { maximumFractionDigits: 1 });
const dayLabel = (day: string) =>
new Date(`${day}T12:00:00Z`).toLocaleDateString('en-GB', { day: 'numeric', month: 'short' });
const sourceLabel: Record<string, string> = {
spotify: 'Spotify',
lastfm: 'Last.fm',
youtube: 'YouTube',
local: 'local files'
};
const now = Date.now();
</script>
<div class="mx-auto max-w-6xl px-4 py-12 sm:px-6 sm:py-16">
<header class="mb-8">
<h1 class="text-4xl font-semibold tracking-tight sm:text-5xl">music</h1>
<p class="mt-3 max-w-2xl text-[var(--site-muted)]">
Everything I listen to, scrobbled from Spotify every minute into my own database (a
self-hosted Last.fm), plus my older Last.fm history.
</p>
</header>
{#if data.connection !== null || data.spotifyStatus}
<aside
class="mb-8 rounded-md border border-[var(--site-border)] p-4 text-sm"
aria-label="Spotify connection (only visible to you)"
>
{#if data.spotifyStatus && data.spotifyStatus !== 'connected'}
<p class="mb-2 text-red-600">Spotify said: {data.spotifyStatus}</p>
{/if}
{#if data.connection}
<p>
Spotify: connected as <strong>{data.connection.user}</strong>
{#if data.connection.lastPollAt}· last poll {relativeTime(
data.connection.lastPollAt,
now
)}{/if}
{#if data.connection.needsReauth}· <strong class="text-red-600">needs reconnecting</strong
>{/if}
</p>
{#if data.connection.lastError}
<p class="mt-1 text-red-600">last error: {data.connection.lastError}</p>
{/if}
{/if}
<a href="/api/spotify/connect" class="mt-2 inline-block underline" data-sveltekit-reload
>{data.connection ? 'reconnect Spotify' : 'connect Spotify'}</a
>
</aside>
{/if}
{#if data.listening}
<div class="mb-8 max-w-md">
<ListeningCard listening={data.listening} href={data.listening.url ?? '/trackers/music'} />
</div>
{/if}
<nav class="mb-6 flex flex-wrap gap-2" aria-label="Period">
{#each ranges as [value, label] (value)}
<a
href="?range={value}"
class="rounded-full border px-3 py-1 text-sm {s.range === value
? 'border-[var(--site-text)] font-medium'
: 'border-[var(--site-border)] text-[var(--site-muted)] hover:text-[var(--site-text)]'}"
aria-current={s.range === value ? 'page' : undefined}
data-sveltekit-noscroll>{label}</a
>
{/each}
</nav>
{#if s.totals.plays === 0}
<div
class="rounded-md border border-dashed border-[var(--site-border)] p-10 text-center text-[var(--site-muted)]"
>
<Music size={28} class="mx-auto mb-3" aria-hidden="true" />
No plays {s.range === 'all' ? 'yet' : 'in this period'}. Scrobbling starts as soon as Spotify
is connected.
</div>
{:else}
<dl class="mb-10 grid grid-cols-2 gap-3 sm:grid-cols-4">
<div class="stat">
<dt>plays</dt>
<dd>{s.totals.plays.toLocaleString('en-GB')}</dd>
</div>
<div class="stat">
<dt>hours</dt>
<dd>{fmtHours(s.totals.hours)}</dd>
</div>
<div class="stat">
<dt>artists</dt>
<dd>{s.totals.artists.toLocaleString('en-GB')}</dd>
</div>
<div class="stat">
<dt>streak</dt>
<dd>
{s.streak.current} day{s.streak.current === 1 ? '' : 's'}
<span class="block text-xs font-normal text-[var(--site-muted)]"
>longest {s.streak.longest}{#if s.streak.longestStart && s.streak.longest > 1}
({dayLabel(s.streak.longestStart)}–{dayLabel(s.streak.longestEnd ?? '')}){/if}</span
>
</dd>
</div>
</dl>
{/if}
<section class="mb-12" aria-labelledby="daily-heading">
<h2 id="daily-heading" class="mb-3 text-xl font-semibold tracking-tight">
hours per day <span class="text-sm font-normal text-[var(--site-muted)]"
>(last {s.daily.length} days)</span
>
</h2>
<div class="flex h-32 items-end gap-px rounded-md border border-[var(--site-border)] p-2">
{#each s.daily as d (d.day)}
<div
class="flex-1 rounded-sm bg-[#1db954]/80"
style="height: {Math.max(d.hours > 0 ? 3 : 0, (d.hours / maxHours) * 100)}%"
title="{dayLabel(d.day)}: {fmtHours(d.hours)} h, {d.plays} plays"
></div>
{/each}
</div>
<div class="mt-1 flex justify-between text-xs text-[var(--site-muted)]">
<span>{dayLabel(s.daily[0]?.day ?? '')}</span><span>today</span>
</div>
</section>
<div class="grid gap-10 lg:grid-cols-3">
{#each [{ id: 'artists', title: 'top artists', rows: s.topArtists.map( (a) => ({ ...a, sub: `${fmtHours(a.hours)} h` }) ) }, { id: 'tracks', title: 'top tracks', rows: s.topTracks.map( (t) => ({ ...t, sub: t.artists }) ) }, { id: 'albums', title: 'top albums', rows: s.topAlbums.map( (a) => ({ ...a, sub: a.artist }) ) }] as list (list.id)}
<section aria-labelledby="{list.id}-heading">
<h2 id="{list.id}-heading" class="mb-3 text-xl font-semibold tracking-tight">
{list.title}
</h2>
{#if list.rows.length === 0}
<p class="text-sm text-[var(--site-muted)]">nothing yet</p>
{:else}
<ol class="space-y-2">
{#each list.rows as row, i (i)}
<li class="flex items-center gap-3">
<span class="w-5 shrink-0 text-right text-sm text-[var(--site-muted)]">{i + 1}</span
>
{#if row.image}
<img
src={row.image}
alt=""
width="40"
height="40"
loading="lazy"
class="size-10 shrink-0 rounded object-cover {list.id === 'artists'
? 'rounded-full'
: ''}"
/>
{:else}
<span class="size-10 shrink-0 rounded bg-[var(--site-border)]"></span>
{/if}
<span class="min-w-0 flex-1">
{#if row.url}
<a
href={row.url}
rel="external"
class="block truncate font-medium hover:underline">{row.name}</a
>
{:else}
<span class="block truncate font-medium">{row.name}</span>
{/if}
<span class="block truncate text-sm text-[var(--site-muted)]">{row.sub}</span>
</span>
<span class="shrink-0 text-sm text-[var(--site-muted)]">{row.plays}</span>
</li>
{/each}
</ol>
{/if}
</section>
{/each}
</div>
<section class="mt-12" aria-labelledby="recent-heading">
<h2 id="recent-heading" class="mb-3 text-xl font-semibold tracking-tight">recent plays</h2>
{#if s.recent.length === 0}
<p class="text-sm text-[var(--site-muted)]">nothing yet</p>
{:else}
<ul class="divide-y divide-[var(--site-border)]">
{#each s.recent as play (play.id)}
<li class="flex items-center gap-3 py-2">
{#if play.image}
<img
src={play.image}
alt=""
width="36"
height="36"
loading="lazy"
class="size-9 shrink-0 rounded object-cover"
/>
{:else}
<span class="size-9 shrink-0 rounded bg-[var(--site-border)]"></span>
{/if}
<span class="min-w-0 flex-1">
{#if play.url}
<a href={play.url} rel="external" class="block truncate hover:underline"
>{play.track}</a
>
{:else}
<span class="block truncate">{play.track}</span>
{/if}
<span class="block truncate text-sm text-[var(--site-muted)]">{play.artists}</span>
</span>
<time datetime={play.at} class="shrink-0 text-sm text-[var(--site-muted)]"
>{relativeTime(play.at, now)}</time
>
</li>
{/each}
</ul>
{/if}
{#if s.sources.length > 1}
<p class="mt-4 text-xs text-[var(--site-muted)]">
sources: {s.sources
.map((x) => `${sourceLabel[x.source] ?? x.source} ${x.plays.toLocaleString('en-GB')}`)
.join(' · ')}
</p>
{/if}
</section>
</div>
<style>
.stat {
border: 1px solid var(--site-border);
border-radius: 0.375rem;
padding: 0.75rem 1rem;
}
.stat dt {
font-size: 0.875rem;
color: var(--site-muted);
}
.stat dd {
font-size: 1.5rem;
font-weight: 600;
}
</style>