houses: one row per listing when a hand-added link is scraped later
All checks were successful
Deploy production / deploy (push) Successful in 3m0s
All checks were successful
Deploy production / deploy (push) Successful in 3m0s
The nightly scraper and worker API now take over an existing row (e.g. source=manual) whose link is the same listing (normalized URL, or the Pararius key / Funda, Kamernet, Vesteda id in it) instead of inserting a copy; typed values are kept where the scrape has none. house-listing update/add give manual rows with such links the site key unless another row has it, update accepts --source/--source-id, and lookups by link ignore tracking params. scripts/house-rekey.ts backfills existing manual rows.
This commit is contained in:
@@ -241,7 +241,7 @@ try {
|
||||
const completed = result.completed && result.listings.length > 0;
|
||||
const ingest = await ingestListings(sql, name, result.listings, { runStartedAt, completed });
|
||||
console.log(
|
||||
`${name}: ${result.listings.length} seen, ${ingest.inserted.length} new${ingest.seeded ? ' (first run: seeded, no alerts)' : ''}, ${ingest.updated} updated, ${ingest.removed} gone`
|
||||
`${name}: ${result.listings.length} seen, ${ingest.inserted.length} new${ingest.seeded ? ' (first run: seeded, no alerts)' : ''}, ${ingest.updated} updated${ingest.adopted ? ` (${ingest.adopted} hand-added rows taken over)` : ''}, ${ingest.removed} gone`
|
||||
);
|
||||
if (!ingest.seeded) for (const row of ingest.inserted) fresh.push({ ...row, source: name });
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user