mcpx testing: shard the suite by measured time, and a rule for when to stop splitting #1994

Relocated from dezren39/darwin-nix#139 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #95

Summary

Decide how the suite is split across matrix legs and how the split is kept balanced as tests are added. Today the whole suite is one job, and one package (internal/e2e, 100.7 s, strictly serial) is three quarters of it. The notes propose: split unit from integration, shard the longer one into logical groups, repeat on the longest shard, then fall back to alphabetical ranges balanced by time, and stop when setup/teardown dominates. This issue makes that concrete, gives the stop rule as arithmetic, and recommends a timing-file-driven splitter over hand-tuned alphabetical ranges -- while keeping the one real advantage of the alphabetical idea (a shard you can reproduce locally with one -run).

Original notes

• testing
  • make a comprehensive github action ci.yaml workflow (or test.yaml) that is as fast as possible,
    • splitting sets of tests into as small groups as practical, eg split out unit/integration tests into 2 jobs, whichever one takes longer: shard it into 4 logical groups (or however many) based on an appropriate category, then find the longest job at that point and repeat (you dont want to iterate on this forever, at some point start splitting logical units by like, e.g. maybe a-f/g-s/t-z, balance number of letters by time.

• testing
  • make a comprehensive github action ci.yaml workflow (or test.yaml) that is as fast as possible,
    • continue until the jobs can't get much faster considering setup/teardown and then set a ci testing budget which is a hardcoded default input. after the test matrix have a time budget job when subtracts github workflow request time from now. (this catches if the jobs are impacted by matrix size, could lessen by taking initial job end or matrix start time.)

• testing
  • make a comprehensive github action ci.yaml workflow (or test.yaml) that is as fast as possible,
    • repeat this process whenever testing exceeds some time budget. make setup faster, rebalance alpha splits, split groups of tests on the longest shard. maybe flag that never tears down because just killing the box after. anything downloaded should be cached, shallow clone a partial path to mcpx dir, if there's unexplainable job start delays, try to keep all job times balanced, 10 ultrafast ones can give you quick signals but may put pressure on runner infra and cause the longest jobs to last a little longer. all that matters is how long the longest jobs take. job runners have 2 or 4 cores, consider if theres lengthy tests which are '1' test but re-exec N times 'in parallel': they will only run 1-3 at a time and so 30 cases could take up to 10 tests worth of time even in parallel and may need to be examined. we're using go so im not too worried about time budget but its best to get this early. main focus is strong tests as each step of the lifecycle.

Current state

From run 36673523537 (see #135 for the full step table); the post-#182 run 36693030957 at 05c78b2 is in brackets:

internal/e2e by file (rg -c '^func Test'; at 05c78b2 in brackets):

Per-test timings do not exist yet: the log has package totals only (no -v, no -json). 199 tests in 100.7 s is ~0.5 s each on average (209 in 108.9 s at 05c78b2, the same rate); the distribution is unknown. For contrast, the ~520 unit tests take ~15 s together, ~0.03 s each -- a 17x gap (from #166). 62 of the e2e tests start a JavaScript runtime (53 "exec" calls in e2e_test.go, 9 in exec_test.go, 3 in runtime_test.go; unchanged at 05c78b2), so one e2e test is commonly 3-4 processes: the CLI, a daemon, a fake MCP server and deno/bun/node.

Parallelism today:

• No t.Parallel() anywhere in the module. go help testflag: -parallel n "Allow[s] parallel execution of test functions that call t.Parallel ... By default, -parallel is set to the value of GOMAXPROCS" -- so it currently does nothing.
• go help build: -p n is "the number of programs, such as build commands or test binaries, that can be run in parallel. The default is GOMAXPROCS". So go test ./... runs up to 4 package binaries at once on the runner, which is why every unit package finished inside e2e's 100 s -- but e2e itself is one binary running 199 tests one after another.
• Runner: this repo is public, so ubuntu-latest = 4 vCPU / 16 GB; a private repo gets 2 vCPU / 8 GB; ubuntu-slim is a 1 vCPU container; macos-latest is 3 vCPU M1 (runner reference). The notes' "2 or 4 cores" is right; which one depends on repo visibility.
• Account limits on the Free plan: 20 concurrent standard-runner jobs (5 macOS), shared by every workflow in the account; 256 jobs per matrix (limits).

Tests that fan out internally ("1 test that re-execs N times in parallel"):

Each mcpx run starts Deno/Bun/Node, which is CPU-heavy at startup, so on 4 vCPU a 6-way fan-out runs roughly 3-4 at a time and on 2 vCPU roughly 2 -- the notes' "30 cases could take up to 10 tests worth of time" is the right intuition. These are also the tests that will contend worst once other tests run in parallel. There are also 7 fixed time.Sleep waits in e2e (100-500 ms each, e.g. e2e_test.go:149,403) that cost wall time regardless of cores.

The procedure, made concrete

1. Measure. Every CI run of e2e uses go test -json (keep human output via gotestsum or a jq filter) and uploads per-test Elapsed as an artifact. The final job (Update dependency vitest to v5 [SECURITY] #140) merges them into one timing file per main run.
2. Unit vs integration. One unit job (go test $(go list ./... | grep -v /internal/e2e)), and e2e separately. The unit job's test time is ~9 s (tui); it will never be the long pole unless something changes drastically.
3. Shard e2e (the approaches below).
4. Repeat on the longest job -- which after step 3 will likely be the nix job, not a test shard (🎨 Palette: Search panel accessibility improvements #143 predicts this). That is the runbook in Update dependency vitest to v5 [SECURITY] #140.
5. Stop by the rule below.

Stop rule

Per-leg fixed cost today, from the step table: queue 2-3 s (occasionally ~40 s) + set up 3.4 s + checkout ~2 s + toolchains ~18-26 s (setup-go 10.9 s alone) + compile and binary build (part of the ~31 s, less with a cache) + post 2 s + completion ~3 s ≈ 30-45 s. With T = 100 s of e2e test time and N equal shards, the longest leg is ≈ fixed + T/N:

Each extra shard saves T/N - T/(N+1) and costs one more runner (and more pressure on the 20-job limit). Stop when T/N is below the fixed cost (going from 4 to 8 legs saves 12 s for 4 more runners), or when the leg is no longer the longest job. Today that means N ≈ 3-4. Reducing the fixed cost (the working Go cache, shipping a compiled e2e.test, #135 decision 7) moves the optimum up; t.Parallel (below) moves T down.

"Many tiny jobs" vs "all that matters is the longest job": more legs are free until they queue. With lint (~3 legs), unit, nix, plugin, docs-examples and #97's environment legs, a 4-shard e2e already puts ~12 jobs in one run; two concurrent runs (a PR and a main push) reach 20 and the 21st waits. Queue time is invisible in the per-job durations and only shows in the run's wall clock -- the budget job reports it separately (#140).

Approaches

A -- hand-tuned alphabetical ranges (the notes' fallback), least recommended as the end state

-run '^Test[A-F]', '^Test[G-S]', '^Test[T-Z]', with boundaries moved by hand when the budget is exceeded.

• Pro: no tooling; anyone can reproduce a shard locally from its name; stable membership.
• Con: every test starts with Test and the next letter follows English, not runtime -- clusters like TestRun..., TestExec..., TestConcurrent... cannot be split by a range boundary without going to two letters (^TestC[a-n]), and after a few rebalances the regexes are unreadable. Balance drifts with every added test, and each rebalance is a workflow edit and a PR -- exactly the "iterate on this forever" the notes warn about. Balancing "number of letters by time" requires the per-test timings anyway, and once you have them, bin-packing uses them better than contiguous ranges can.

B -- logical shards by feature (file), a static list

One shard per feature file: exec, consumer, v1ops, proto, settings, logs, runtime, multiconfig, plus the rest.

• Pro: a red leg names a feature; stable; readable.
• Con: badly unbalanced by construction -- e2e_test.go holds 64 % of the tests, so the "rest" shard is roughly two thirds of the time and needs splitting again. Go cannot select tests by file, so each shard's -run regex is a generated list of names (grep '^func Test' file). To make B balance, e2e_test.go would have to be broken into feature files first -- a worthwhile readability refactor on its own (one file of 2,977 lines and 129 tests at 05c78b2), but not something to do for CI balance.
• Variant (from pkgs/graph Go build fails on sshmarshal/x/crypto symbol collision #166): split internal/e2e into subpackages (internal/e2e/exec, .../proto, .../v1ops, ... with newEnv and the fixtures lifted into a shared helper package). Each subpackage is its own test binary, so shards need no regex, survive renames, cannot silently select zero tests, and compile in parallel; go test ./internal/e2e/... still means "all e2e". Same arithmetic limit as B: on today's file boundaries the largest shard is still e2e_test.go.

C -- timing-file-driven splitter (recommended)

A small Go program in the repo (e.g. pkgs/mcpx/internal/ci/shard or scripts/ci/shard.go, run with go run; the toolchain is already there):

1. lists tests (grep '^func Test' over internal/e2e/*_test.go is free; go test -list . is exact but compiles the test binary first);
2. reads the latest timing file -- the artifact of the last successful main run (via gh run download / actions/download-artifact with run-id), falling back to a committed timings.json, falling back to equal weights;
3. gives tests without a timing the median;
4. computes N = clamp(ceil(total / target_leg_seconds), 1, max_shards), and assigns tests to shards by longest-processing-time greedy bin-packing (sort descending, put each into the currently lightest shard);
5. emits the matrix JSON: [{"name":"e2e-1","run":"^(TestA|TestB|...)$"}, ...], and each leg prints its exact go test -run '...' line into the job summary.
6. makes each leg assert it ran the number of tests it was assigned. A -run regex that matches nothing makes go test print "testing: warning: no tests to run" and exit 0 -- a shard that ran nothing is green (from pkgs/graph Go build fails on sshmarshal/x/crypto symbol collision #166).
7. records subtests as their own units: TestGeneratedClientWorksOnEveryRuntime and TestSessionIsolationHoldsOnEveryRuntime (internal/e2e/runtime_test.go:13,45) are one func Test each but three runtimes' worth of time, so a packer that weighs by func Test under-counts them 3x (from pkgs/graph Go build fails on sshmarshal/x/crypto symbol collision #166 / Phase 2: production ingest enforces only p2panda-auth membership #168).

Why it beats A: balance within a few percent regardless of naming, rebalancing is automatic on every run (no workflow edits), and "split the longest shard" becomes changing one number (target_leg_seconds). The alphabetical idea's real benefit -- a shard you can reproduce locally -- is kept by printing the regex, and the #99 policy can say "rerun the failing leg with the line in its summary". Honest costs: ~100-150 lines to own; shard membership changes between runs (fine, since the summary names the tests); timings are noisy (use the median of the last few main runs, not the last one); a very long -run regex is ugly but Go handles thousands of alternatives fine.

Existing tools considered: gotesplit splits by test count, not time; CircleCI's tests split --split-by=timings is CircleCI-only; gotestsum records -json but does not split. Nothing off the shelf does LPT from Go JSON timings on GitHub Actions without adopting a larger framework, so a small in-repo program is the honest choice.

Hybrid worth considering: keep the -run names meaningful by packing whole files first (B's readability) and only splitting e2e_test.go by LPT. Adds complexity for readability only; decide after the first timings.

Beyond sharding: t.Parallel in e2e (the biggest single win; #52 no longer blocks it)

199 (now 209) serial tests that mostly wait on process start, sockets and runtime startup are the textbook case for t.Parallel: on one 4 vCPU runner it could plausibly cut e2e 2-3x with zero extra setup cost, and it composes with sharding (fewer legs needed). What stood in the way, and what still does:

• chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52: leaked daemons multiply under parallel runs, and the cleanup error is discarded. Fixed in Phase 5: tag event bodies must travel with the event #182: cleanup runs mcpx stop --all through the info files and logs its error (internal/e2e/e2e_test.go:71-85), and every test daemon idles out after 60 s (:68, :92).
• Update dependency htmx.org to v1.9.12 #138: newEnv still starts from os.Environ() (e2e_test.go:52), so an exported MCPX_DAEMON_ENDPOINT or MCPX_SOCKET sends every test -- every parallel test at once -- to one outside daemon. Fix before parallelising.
• Shared state audit: no t.Setenv/os.Setenv in e2e (good -- t.Setenv forbids t.Parallel), and each newEnv isolates state/cache/config dirs (internal/e2e/e2e_test.go:35-87). Unknown: fixed ports, the stub registry, anything in $HOME, and whether a long t.TempDir() path pushes two tests' sockets into the shared private runtime dir (internal/daemon/paths.go:103-133; hashed per state dir, so a collision is unlikely but the directory is shared -- chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to fd4ce63 #88).
• Process count, not test count, is the unit of contention: each parallel test is 3-4 processes (above), so -parallel 8 on 4 vCPU is 24-32 processes and the 90 s per-command timeout (e2e_test.go:113) stops being generous. Measure -parallel at 2, 4, 6, 8 and record the chosen value with its reason, rather than taking GOMAXPROCS (from pkgs/graph Go build fails on sshmarshal/x/crypto symbol collision #166 / Phase 2: production ingest enforces only p2panda-auth membership #168).
• The fan-out tests above should stay serial (or get their own leg) so they do not starve the rest.
• Update dependency @playwright/test to v1.64.1 #79 (faster CLI/exec, no self-socket) reduces per-test time directly; every e2e test pays process start.

Recommended order: #143 (unit/e2e split, first static shards, timing capture; the cache line already landed in #182) -> C with N = 3-4 -> #138's allowlist, then t.Parallel in e2e at a measured -parallel, re-measure, reduce N.

Unexplained job-start delays

Measure, do not guess: the jobs API gives created_at (queued), started_at, and per-step times for every job (#140 reports max and sum of queue delay per run). Observed so far: 2-3 s in 9 of the last 10 runs, 37 s once (run 36663358199). If queue delay grows with leg count, merge the smallest legs (lint legs are the obvious candidates) before adding shards.

Open questions

1. Per-test time distribution of e2e: is it flat (~0.5 s each) or dominated by a few slow tests? Decides whether LPT or "fix the ten slowest tests" comes first. (🎨 Palette: Search panel accessibility improvements #143 captures it.)
2. Timing source: last main run's artifact (fresh, no commits) vs a committed file (reviewable, deterministic, needs a bot or manual refresh). Recommendation: artifact with committed fallback.
3. target_leg_seconds default: tie to the budget (Update dependency vitest to v5 [SECURITY] #140) or set independently?
4. Should unit packages join the packer (as package-granularity items) or stay one leg? Probably one leg until units exceed ~30 s.
5. Split e2e_test.go into feature files regardless of CI? (Readability win; separate issue if wanted.)
6. Does t.Parallel in e2e need a per-test daemon idle-exit (chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52 fix 1) first, or only the stop-error fix? Answered by Phase 5: tag event bodies must travel with the event #182, which landed both. The remaining precondition is Update dependency htmx.org to v1.9.12 #138.
7. A per-shard -timeout? A hung shard currently burns the default 10 minutes and Update dependency vitest to v5 [SECURITY] #140 would report it as a budget breach rather than a hang (from pkgs/graph Go build fails on sshmarshal/x/crypto symbol collision #166).
8. internal/tui (8.6 s) stays in the unit leg: it is the second-largest package but does not justify a leg's fixed cost (from pkgs/graph Go build fails on sshmarshal/x/crypto symbol collision #166).

Ease · Impact · Related

Ease M · Impact high · Depends on / related: #95, #135 (matrix from init), #140 (timing merge, runbook), #143 (first measurements), #138 (environment allowlist -- the remaining precondition for t.Parallel), #52 (closed, fixed in #182 -- no longer blocks t.Parallel), #88 (shared socket fallback dir), #79 (faster CLI/exec shortens every e2e test), #97 / #134 (environment legs share runner capacity), #130 (the changed-tests -race -shuffle -count=N PR job runs beside the matrix and must not become the longest leg), #99 / #127 (agents rerun one shard from its printed -run line, not the suite), #93 (seed; -shuffle interacts with ordering, not with membership).

Security/auth deferred; see tracker.

Acceptance criteria

• The suite is sharded by measured time per the body's rule, shard timings are recorded on this issue, and the stop-splitting rule is documented in the workflow/repo.
• The work described in the Summary/Scope above is delivered via a branch + PR (no direct push to main); verification runs in GitHub Actions CI, not on the sandbox.
• Evidence (CI links, timings, or probe output as applicable) is recorded on this issue before it is closed.