Repository navigation
id(ci): flake-check builds can receive a corrupted source copy — run 40617526037 failed 7 checks with parse errors absent from the git tree #1569
Description
Problem
CI run 40617526037 (PR #1563, head 56f010e) failed 7 pkgs/id flake checks (rustfmt-check, cargo-check, test-shard-1/2/3, crash-gate-sampled, crash-gate-ingest) with symptoms of a corrupted source copy inside the build, not the committed code:
- rustfmt job 122589807600 log shows Rust parse errors in the build copy of
pkgs/id/src/directory.rs:error[E0765]: unterminated double quote stringat :2925 andprefix 'com' is unknownat :2896 (normalize_email("ok@example.com")). - The git content at the PR head (verified via the GitHub contents API at 56f010e, 2026-10-28T02:44Z) has those exact lines intact and valid; the PR diff is that single file and its committed form parses.
- All failures are confined to the one run; other workflows on the same head (nix ci) passed.
So the source snapshot the nix builds received differed from the git tree — torn/corrupted in a way that split string literals. Every check that compiled that file failed the same way, which reads exactly like several independent code failures until the logs are compared.
Impact
- Blocks PR id(directory): drop + truncate torn final journal line in replay (#1521) #1563 (task 1622 / issue id(directory): a torn final line in directory.jsonl bricks the directory — Directory::replay refuses the crash-torn tail that ArtifactLog (loaded in the same open) drops and truncates #1521) until the run completes and failed jobs can be re-run.
- Wastes reviewer/worker cycles root-causing phantom parse errors; similar in spirit to the Magic Nix Cache HTTP 418 flake seen on PR for id(web): a marked span starting or ending with a space loses its mark on reopen when a word character touches it — saved   form cannot reparse #1418 (task 1485, 2026-10-28).
Acceptance
- Identify where the build's source copy can diverge from the git tree (flake source filtering / cache / checkout path) for run 40617526037, or catch the divergence mechanically (e.g. a source-tree hash check early in the id workflow that fails fast with a clear "source snapshot mismatch" message instead of 7 downstream parse/test failures).
- A re-run of the failed jobs on the same head passes without any code change, OR the divergence mechanism is found and fixed.
Filed by repo-queue-worker lane A (task 1622 resume, 2026-10-28).
Acceptance criteria
- The divergence point for run 40617526037 is identified — where the flake build's source copy can differ from the git tree (flake source filtering / cache / checkout path) — OR a mechanical guard exists (e.g. a source-tree hash check early in the id workflow) that fails fast with a clear "source snapshot mismatch" message instead of 7 downstream parse/test failures.
- A re-run of the failed jobs on the same head passes without any code change, OR the divergence mechanism is found and fixed.
- Root-cause findings are recorded on this issue either way, so the next phantom-parse failure is recognizable on sight.
Activity
developing-today commentedon Oct 28, 2026
CODE-613
dezren39 commentedon Oct 28, 2026
Intake review
Related: PR #1563 (the victim PR — a single-file diff to
pkgs/id/src/directory.rswhose committed form parses; still open, blocked by this) · issue #1521 / queue task 1622 (the work PR #1563 serves; the task's resume notes name this same corruption) · the Magic Nix Cache HTTP 418 flake on the PR for #1418 (task 1485) as a sibling CI-infra phantom.Notes: Still live — verified this run: no merged PR resolves it; PR #1563 is open and its head is the run in question. The issue arrived with a
## Acceptancesection; the appended## Acceptance criteriaformalizes it and adds the record-the-root-cause criterion. Evidence quality is high (run/job IDs, git-content comparison at head56f010e9).Risks / suggested approach: CI-investigation class: start from the run's checkout/cache path rather than the code; if the mechanism cannot be reproduced, the hash-check guard is the durable deliverable. Effort classed
effort/m— infra root-causing is open-ended, and a one-line fix after a long hunt is the expected shape.— repo-fleet-monitor intake pass, 2026-10-28T03:50Z
dezren39 commentedon Nov 1, 2026
agent-note:v1 — queue task 2608 (lane E) — triage verdict: CONFIRMED one-off divergence; root cause unproven; routed to the guard branch
Verified 2026-11-01:
idrun for that head (status completed/cancelled);nix ciand Auto Assign on the same head both succeeded — the torn copy was confined to the id workflow's source path on that runner..github/workflows/id-check.ymland pkgs/id on main (0ea0a0f) finds no source-tree hash or snapshot-mismatch guard.Route: ci_investigation packet enqueued as queue task 5020: rank the divergence candidates from the run logs + workflow definition and record the findings here, and add the early source-snapshot integrity guard (digest at checkout, verify the build's copy, fail fast with "source snapshot mismatch" naming the differing files), verified clean-pass and mutated-fail.
64 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up (task 5020 resumed, lane B)
error: '/home/runner/work/code/code' is a shallow Git repository, so 'revCount' is not available(Determinate Nix getFlake on the shallow actions/checkout). Same failure on run 41301886497 (PR fix(ci): derive id flake source path via getFlake outPath (#1569) #3524 head).&shallow=1to the getFlake URL in both Verify steps (tells nix not to require revCount for the source copy), verify the expression + guard both ways on the laptop against a shallow clone, merge, and confirm the Verify step passes on the next main id run. Fallback if shallow=1 is not honoured:fetch-depth: 0on the id-check checkouts.Merge pull request #3675 from developing-today/queue/5020-1569-verify…
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done — guard wiring FIXED and verified in CI (queue task 5020, lane B)
c35ab37de) adds&shallow=1to thegit+file://getFlake URL in both Verify steps of.github/workflows/id-check.yml. Root cause of the round-2 failure: actions/checkout is shallow and Determinate Nix refuses getFlake revCount on a shallow repo (revCount is not available), so the Verify step failed on every main run (41301886497, 41303442327) while Snapshot succeeded.idrun 41304099604 (headc35ab37de): "Verify build source copy against checkout snapshot" = success, and the job proceeded to "Run full nix flake check" — the first main run where the guard actually compares the build source copy.--depth 1clone: clean verify passes (477 files); a mutated store copy fails exit 1 withsource snapshot mismatchnamingsrc/directory.rs. Codex review of PR fix(ci): shallow=1 on the id source-snapshot getFlake derivation (#1569) #3675 completed with no findings.