codeSplitter shadow --check: run it over every git-only merge and triage the failures #1141

Activity

1. QuinnsCode

   self-assigned this

   on Oct 12, 2026
2. QuinnsCode commented on Oct 12, 2026

   

   QuinnsCode

   on Oct 12, 2026

   OwnerAuthor

   More actions

   Status from the PR agent, 2026-10-12: a first attempt at this was stopped by the owner.

   A worker started the triage run (shadow.ts --check over ~90 PRs, 3 at a time). Every fetch under the sandbox's empty HOME asked macOS's Keychain through Apple's system credential.helper = osxkeychain, and each one popped a "Keychain Not Found" dialog, which made the Mac unusable. #1152 (merged) clears git's credential helper and terminal prompts in the shadow sandbox, checkEnv and trial-github.ts.

   Nothing was triaged yet: no results were saved, and no commits or PR were made. The one uncommitted change, kept here so the worktree can go, adds --jobs N to run that many checks at once, with a progress line per merge:

   diff --git a/apps/codesplitter/scripts/shadow.ts b/apps/codesplitter/scripts/shadow.ts
   index e44e150..c6fd2c6 100644
   --- a/apps/codesplitter/scripts/shadow.ts
   +++ b/apps/codesplitter/scripts/shadow.ts
   @@ -18,7 +18,7 @@
    // Each clean result is also checked against what landed on main: the PR's own
    // resolution, which a person or agent made and CI passed.
    //
   -//   bun scripts/shadow.ts [--limit 1000] [--pr 893,886] [--json] [--check]
   +//   bun scripts/shadow.ts [--limit 1000] [--pr 893,886] [--json] [--check [--jobs 3]]
    //
    // `--check` (#1043, needs `--pr`: an install per merge) builds and tests each
    // merge with git-only files: git's merge of the two sides (merge-tree), ours
   @@ -28,7 +28,8 @@
    // and only the tests that fail on ours and pass there count (`fail`); failing
    // on both (`same`) is the sandbox or a native build, not the merge.
    // Tests run in a throwaway folder with an empty HOME and none of this shell's
   -// env, so nothing reaches a live account.
   +// env, so nothing reaches a live account. `--jobs` checks that many merges at
   +// once (default 1); each prints its result to stderr as it finishes.
    //
    // The PR list (number, head, merge commit) comes from `gh pr list`, read only;
    // `--from prs.json` reads the same fields from a file instead. A PR whose head
   @@ -280,6 +281,7 @@ if (import.meta.main) {
      const tally: Record<Verdict | "prs" | "skipped" | "merges" | "files", number> = { agree: 0, both: 0, "git-only": 0, "ours-only": 0, differ: 0, guard: 0, lock: 0, prs: 0, skipped: 0, merges: 0, files: 0 };
      const found: { label: string; path: string; s: Shadow; check?: Check }[] = [];
      const checks: Record<Check["result"], number> = { pass: 0, fail: 0, same: 0, untested: 0 };
   +  const pending: { j: Join; gitOnly: (typeof found)[number][] }[] = [];
      for (const pr of prs) {
        if (only && !only.includes(pr.number)) continue;
        tally.prs++;
   @@ -303,13 +305,20 @@ if (import.meta.main) {
            found.push({ label: j.label, path, s });
            if (s.verdict === "git-only") gitOnly.push(found.at(-1)!);
          }
   -      if (!checking || !gitOnly.length) continue;
   -      console.error(`checking ${j.label}: ${gitOnly.map((f) => f.path).join(", ")}`);
   +      if (checking && gitOnly.length) pending.push({ j, gitOnly });
   +    }
   +  }
   +  // A few merges at a time: most of each one's 3 minutes is install and tests, not this process.
   +  let next = 0, done = 0;
   +  await Promise.all(Array.from({ length: Math.max(1, Number(arg("jobs") ?? 1)) }, async () => {
   +    while (next < pending.length) {
   +      const { j, gitOnly } = pending[next++]!;
          const c = await check(j.main, j.branch, j.landed, new Map(gitOnly.map((f) => [f.path, f.s.ours.text!])));
          checks[c.result]++;
          for (const f of gitOnly) f.check = c;
   +      console.error(`[${++done}/${pending.length}] ${j.label}: ${c.result}${c.failed?.length ? ` (${c.failed.join("; ")})` : ""}`);
        }
   -  }
   +  }));
      if (process.argv.includes("--json")) console.log(JSON.stringify({ tally, ...checking && { checks }, found }, null, 2));
      else {
        console.log(`${tally.merges} merges from ${tally.prs - tally.skipped} PRs (${tally.skipped} skipped: not in this clone), ${tally.files} files both sides changed`);

   Before re-running: start from current main (which has #1152). Try a single PR first (--pr <n> --check --jobs 1) and confirm no Keychain dialog appears, then widen it.
3. QuinnsCode

   added a commit that references this issue on Oct 30, 2026

   codeSplitter: shadow --check takes --jobs to check merges in parallel (…

   ...

   5fb56fd
4. QuinnsCode commented on Nov 8, 2026

   

   QuinnsCode

   on Nov 8, 2026

   OwnerAuthor

   More actions

   First pass: 5 PRs, 6 merges, on Bob (2026-11-08)

   Keychain first. #1152's NO_KEYCHAIN is on main and in sandbox(). Under that env, git's helper list is the system osxkeychain followed by an empty command-line entry that resets it. git credential fill for github.com under GIT_TRACE stops at "terminal prompts disabled", and osxkeychain never starts. No dialogs came up. The run went to Bob (the owner's call: heavy work goes to the builder), one merge at a time.

   What Bob needed: his slots have main's merge commits but not the deleted PR heads, and gh isn't signed in there. So the PR list goes over as a file (--from), and the heads come from GitHub's public refs/pull/N/head, fetched in the same command with the no-keychain env (objects only: no refs or branches made).

   merge	git-only files	result
   #3901 ce2242b	eqosys qollab/party.ts, packages/util/src/eqosys.ts	pass
   #3894 bd5c200	two-fronts src/main.ts	pass
   #3731 7834ec4	codesplitter api.ts, openapi.ts, test/agent-routes.test.ts	fail, noise (below)
   #3685 15109ee	mythic-masters main.ts, world/view.ts	pass
   #2719 9728845	two-fronts README.md, war/rules.ts	pass
   #2719 1b9e84d	two-fronts README.md	pass

   Triage of #3731's fail: a note, not a fixture. The test that failed on ours and passed on what landed is swarm-stage.test.ts "Next walks every scene in order". That test walks the swarm story in a real browser view (Bun.WebView), it failed in 4 ms, and it reads none of the three merged files. It's a flaky browser test in a 1064-test run, not a line merge bug. Three other codesplitter tests failed on both trees ("a file dropped into a thread", "folder mirrors > pulls the folder's commits", "beaming a branch"), so they come from the sandbox on Bob, not the merge.

   Worth adding before the full run: when a test fails on ours and passes on what landed, run that one test once more on ours before calling the merge fail. That would have saved this triage.

   Scale: the last 1000 merged PRs have 81 merges with git-only files, across 59 PRs (up from #1014's sweep). Six merges took 713 s on Bob, so the full run is about 3 hours.

   The full run (from the office Mac, Bob on)

   cd apps/codesplitter
   gh pr list --state merged --limit 1000 --json number,headRefOid,mergeCommit > .shadow-prs.json
   P=3901,3894,3782,3732,3731,3693,3685,3174,3135,3113,3073,3068,3003,2959,2909,2900,2776,2775,2774,2772,2761,2754,2751,2726,2724,2719,2714,2662,2655,2654,2653,2650,2643,2611,2596,2566,2528,2527,2509,2507,2502,2470,2423,2421,2394,2361,2350,2349,2269,2205,2187,2159,2150,2148,2112,2110,2108,2107,2105
   REFS=$(echo $P | tr , '\n' | sed 's|.*|refs/pull/&/head|' | tr '\n' ' ')
   cd ../.. && bun run builder run apps/codesplitter sh -c "export GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=credential.helper GIT_CONFIG_VALUE_0= GIT_TERMINAL_PROMPT=0; git fetch -q https://github.com/QuinnsCode/RustyBuns.git $REFS && bun scripts/shadow.ts --from .shadow-prs.json --pr $P --check --json" > check.out 2>&1

   Builder output lines start with [apps/codesplitter] , so strip that prefix before parsing the JSON. Afterwards, delete .shadow-prs.json. Leaves #1141 open until the full run is triaged.
5. QuinnsCode

   added

   current-needLane: we need this nowLane: we need this now

   on Nov 9, 2026
6. QuinnsCode commented on Nov 9, 2026

   

   QuinnsCode

   on Nov 9, 2026

   OwnerAuthor

   More actions

   The full run: 81 merges over 59 PRs, on Bob (2026-11-09)

   Run for #3946, one PR per bun run builder run (so a hang costs one PR, not the night), Bob on battery the whole way (one suite at a time). 02:16 to 07:42, about 5½ hours.

   One real failure in 81. codeSplitter's merge broke tests that pass as landed exactly once, and that's filed as #4133.

   result	merges
   pass	47
   same (fails as landed too: sandbox/env, not the merge)	18
   untested (no app tests cover the files)	9
   fail	7: 1 real, 6 the check's own blind spot

   Flaky: none came out flaky. check() already reruns a suspect test once, and every fail failed twice.

   The seven fails

   merge	verdict	why
   #2654 17cdbb0 (fm-daw)	real: #4133	both sides added src/ui/net.ts. codeSplitter called it clean but kept one side per line, dropping locked from some status calls and floor from others. Two online tests fail.
   #3732 c754e01 (codesplitter)	check's blind spot: #4134	the merge commit wrote keepIn into thread-do.ts (in neither parent), and the landed agent-routes.ts imports it
   #2726 b3f88ed, b5d6e9a (two-fronts)	check's blind spot: #4134	the merge commit moved yearOf into objectives.ts: Export named 'yearOf' not found
   #2726 04a938c (two-fronts)	check's blind spot: #4134	the merge commit added calm?: true and a Gale check to rules.ts, plus edits to gale.ts, icons.ts and gale.test.ts
   #2959 5e7de7b (mythic-masters)	check's blind spot: #4134	the merge commit retuned balance.json (128 lines; git merged that file cleanly)
   #2159 d46594c (skate-tech-park)	check's blind spot: #4134	the merge commit wrote a step-aside loop into earthworks.ts's spawn search

   A "blind spot" is a merge commit that also carried new work. The tree under check mixes git's merge, codeSplitter's files and landed files, and can't hold together around new code. #4134 asks check() to call those unclear.

   Fixed along the way

   #3113 526313f first came out fail: every kelpie and minotaur GLB test failed. writeMerge read the landed version of a conflicted binary file as text and mangled the model. #4132 copies it byte for byte, and the rerun on that fix gives same.

   The PR list (--limit 1000) no longer reached back to #2205-#2105 (1000 merges ago is now #2263), so those ten came back empty. I rebuilt the list with --limit 1300 and reran them. The next full run needs a bigger --limit, or a list built from the PR numbers.

   Every merge

   pass (47): #3901 ce2242b, #3894 bd5c200, #3782 a4458ea, #3693 6d76bee, #3685 15109ee, #3174 1bb3d4d, #3135 7f0aa96, #3073 e6bcfd2, #3003 f5f2a2c, #3003 9b2dd8b, #3003 3bb7e3d, #3003 0b155cb, #3003 04d4483, #3003 bfc2501, #3003 93ee1dd, #2959 67fcf06, #2909 dd58bd8, #2776 c018119, #2772 5ec34a8, #2726 90f8379, #2726 378d97e, #2719 9728845, #2719 1b9e84d, #2662 a22e2a3, #2655 b7afe5b, #2653 206c63d, #2650 605c6c9, #2643 bd69db3, #2643 605c6c9, #2611 fb04c76, #2528 e8e26f6, #2470 401590a, #2423 087dfac, #2421 f51858a, #2394 2ad1e1b, #2361 da2e52f, #2350 0555854, #2349 088cf76, #2269 e6ec56a, #2150 44357e5, #2150 f9a8fa4, #2148 f9a8fa4, #2112 dec3a03, #2110 9a6c218, #2108 6dc0a14, #2107 a2a0a53, #2105 aae11c2

   same (18): #3731 7834ec4, #3113 526313f, #3068 f6e43db, #2900 836a9e1, #2776 e7ff1d3, #2775 a282abc, #2726 2c998a1, #2596 b92ed8a, #2566 1521e09, #2527 14a66da, #2509 e807525, #2507 20d2fdc, #2502 9a8c3a7, #2269 a9181e5, #2269 e2443d1, #2269 e3ac5c1, #2269 52fcf25, #2148 80941ff

   untested (9): #2774 a52b74c, #2772 fa63104, #2761 0061b1b, #2754 8bb92b1, #2751 8c91ea0, #2724 f1b6826, #2714 a853e61, #2205 e10bd7d, #2187 1c2992f

   fail (7): #3732 c754e01, #2959 5e7de7b, #2726 b3f88ed, #2726 b5d6e9a, #2726 04a938c, #2654 17cdbb0, #2159 d46594c

   .shadow-prs.json is deleted. This leaves #1141 open, for #4133 and #4134.
7. QuinnsCode

   added a commit that references this issue on Nov 9, 2026

   shadow --check: a conflicted binary file comes from what landed byte …

   ...

   bca6fc1