lab-ops-hub: crawl archive R2 key is Date.now() alone — same-millisecond crawls silently overwrite each other's archived full text, both D1 rows point at the second crawl #2503

Bug: lab-ops-hub crawl archive R2 key is Date.now() alone — same-millisecond crawls silently overwrite each other's archived full text, and both D1 rows point at the second crawl's object

POST /api/crawl in pkgs/lab-ops-hub/src/index.ts:399 archives each crawl's full extracted result to R2 under a key derived from the wall-clock millisecond only — no URL, filename, random, or row-id component:

const key = `crawls/${Date.now()}.json`;
await env.FILES.put(key, JSON.stringify({ url: b.url, source, ...result }, null, 2), { ... });
const r = await env.DB.prepare(
  "INSERT INTO crawls (url, source, title, excerpt, links, status_code, r2_key) VALUES (?1,?2,?3,?4,?5,?6,?7)",
).bind(b.url, source, result.title ?? null, ..., key).run();

Unlike the sibling file-upload key (#2478), this collision does not surface: pkgs/lab-ops-hub/schema.sql declares crawls.r2_key TEXT with no UNIQUE constraint (the files table's r2_key is UNIQUE; the crawls table's is not). Two crawls that complete in the same millisecond therefore:

• derive byte-identical keys (crawls/<ms>.json);
• the second FILES.put silently overwrites the first crawl's archived JSON (full text, links, status) in R2;
• both D1 INSERTs succeed, leaving two crawl rows whose r2_key is the same object — the first row's metadata (url/title/excerpt) now describes an archive that actually contains the second crawl's content. The first crawl's full text is unrecoverable.

Millisecond collisions are realistic for this endpoint: scripted/retried crawls, a caller re-submitting after a container timeout (the handler itself falls back and retries via the Worker path), and test loops all land back-to-back crawls within one millisecond — and Date.now() resolution makes the window wider than it looks on coarse clocks.

Verified this run (main 7d1b081c)

Sandbox Python probe (probe dir /tmp/probe-laneA-crawlkey/, script probe.py) that extracts the verbatim key expression from src/index.ts, builds the crawls table from the real schema.sql, and replays the handler's put → insert order for two different URLs at one fixed millisecond:

• Key line found verbatim in index.ts: const key = \crawls/${Date.now()}.json`;`
• crawls table definition contains no UNIQUE.
• Two crawls (https://a.example/ → TEXT-A, https://b.example/ → TEXT-B) at ms 1761782400000: both inserts succeed (ids 1 and 2), distinct keys = 1, R2 objects = 1, and the archive served for row 1's r2_key is TEXT-B, not TEXT-A.

Why it matters

The crawls table + R2 archive are the hub's record of what was crawled: the D1 row keeps only a 500-char excerpt inline, so the R2 object is the only full-text copy. A collision silently swaps the archive behind an existing row — the row looks healthy in /api/crawls, but its archive belongs to a different URL, and the original crawl's text is lost with no error anywhere.

Acceptance criteria

• Two crawls completing in the same millisecond both succeed with distinct R2 keys and two D1 rows whose archives each contain their own crawl's content (e.g. a random/UUID or row-id component in the key, as lab-ops-hub: file-upload R2 key is Date.now()+filename — same-millisecond uploads collide, overwrite the object, then 500 on the UNIQUE r2_key #2478 prescribes for uploads).
• A regression test exercises back-to-back same-millisecond crawls (the key derivation should live somewhere testable, per the lab-ops-hub: the #500 crawl-timeout fix has no regression test — crawlFallback's fetcher is inline in the untestable Worker entry #524 pattern of moving logic out of the Worker entry into src/lib.ts).
• Existing behaviour otherwise unchanged: key prefix crawls/, .json suffix, the archived JSON shape, and the response body.

Dedup

Distinct from #2478 (file-upload key uploads/<ms>-<filename>): that key at least carries the filename, and its collision is caught (as a 500) by the files.r2_key UNIQUE constraint. The crawl key carries nothing but the millisecond and its table has no UNIQUE guard, so the failure mode is silent archive loss, not an error. gh searches (open + closed) for lab-ops-hub crawl / r2_key / Date.now terms return the crawl-validator and timeout issues (#441, #460, #500, #511–#519, #532, #569, #571) and #2478 — none covers the crawl archive key derivation.

— filed by new-issue-creator-code-a (Lane A), 2026-10-29 run