Repository navigation
mcpx testing: a fast, sharded CI with a time budget, linters and auto-format #1901
Description
Relocated from dezren39/darwin-nix#95 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).
Parent: #91
Purpose
CI for mcpx, built to the notes' stated top priority: "the main priority is being short as possible", with its stated focus, "main focus is strong tests as each step of the lifecycle". Those two pull against each other, and this tracker is where the trade is made explicitly rather than by accident.
Where it starts, measured rather than estimated. One workflow, .github/workflows/mcpx.yml, one ubuntu-latest job. Run 36673523537 (push to main), from the step timestamps in gh run view 36673523537 --log:
| Step | Seconds |
|---|---|
| Set up job, checkout, nix-installer, setup-go/deno/bun/node | 32.6 |
go test ./... -count=1 |
134.8 |
nix build .#packages.x86_64-linux.mcpx |
58.8 |
| Post steps + complete | 1.5 |
| Total wall clock | 229 s |
Inside the test step, internal/e2e alone is 100.727 s of 134.8; internal/tui 8.6 s; internal/pool 1.5 s; internal/events 1.05 s; internal/mcpserver 1.03 s; everything else under 0.5 s. The last eight runs sit between 3 m 57 s and 4 m 18 s. So the whole sharding question is, in the first instance, one question: what do you do about e2e, which is 44% of wall clock and 199 of 719 tests.
After #182, at 05c78b2 (run 36693030957, push to main): 4 m 09 s wall; setup 22 s; go test 148 s with internal/e2e at 108.9 s for 209 of 735 tests; nix build 70 s (checkPhase 18.5 s). That run's Go cache was a miss by construction (the first main run with the new key), so the warm-cache number is still owed -- #143 records it.
Three facts about today's setup that this tracker exists to fix, all verified:
The Go cache has never restored.Fixed in Phase 5: tag event bodies must travel with the event #182.actions/setup-go@v7defaultscache: truebut looked forgo.modat the repository root; mcpx's is atpkgs/mcpx/go.mod. The log carried##[warning]Restore cache failed: Dependencies file is not foundand every run re-downloaded every module and compiled cold. Phase 5: tag event bodies must travel with the event #182 addedcache-dependency-path: pkgs/mcpx/go.sum(.github/workflows/mcpx.yml:50); the key now saves. What remains of caching (restore-only on PRs, the Nix store, toolchains) is Update Rust crate tokio to v1.38.2 [SECURITY] #135.treefmt.nixexists and nothing runs it.programs.nixfmtandprograms.rustfmtare enabled,flake.nixexposeschecks.formatting, and no workflow invokes either.treefmt.nix:6also carries# programs.shellcheck.enable = true; # failures-- a linter switched off rather than satisfied, already in the tree, and the precedent any new linter must avoid repeating.- Unit tests run twice per CI run.
package.nixcheckPhasere-runs a list of unit packages insidenix build, costing 16.55 s (18.5 s at 05c78b2) on top of thego teststep. That is either free time to reclaim or a deliberate sandbox check worth keeping; Update Rust crate tokio to v1.38.2 [SECURITY] #135 decision 6 now argues to keep it (the sandbox has no network, which would have caught Update Rust crate axum to 0.6.20 #65) but take it off the critical path, and to fix its drifted package list (12 packages with tests never run there).
What this tracker does not own: the seed itself (#93 -- the init job mints and exports it, but #121/#122 decide what it is), the environment legs that join this matrix (#97 -- nix, container, remote, other OSes), and what an agent runs on its own machine instead of CI (#99).
Children
- Update Rust crate tokio to v1.38.2 [SECURITY] #135 mcpx testing: CI workflow skeleton -- init, matrix, final job; triggers, concurrency, caching, checkout (its cache one-liner landed in Phase 5: tag event bodies must travel with the event #182)
- Update dependency hyperscript.org to v0.9.93 #139 mcpx testing: shard the suite by measured time, and a rule for when to stop splitting
- Update dependency vitest to v5 [SECURITY] #140 mcpx testing: a CI time budget in the final job, and the over-budget runbook
- Bump dezren39/nix flake input to 3499ed3 — opencode now defaults to v2 #141 mcpx testing: every formatter and linter in the CI matrix, pinned by the flake (and run the plugin's bun test)
- ⚡ Bolt: Optimize search match counting performance #142 mcpx testing: format.yaml -- auto-format after a green run; PR head vs main
- 🎨 Palette: Search panel accessibility improvements #143 mcpx testing: first CI iteration -- unit + e2e shards + lint + nix build, and a recorded baseline
Proposed order
- Update Rust crate tokio to v1.38.2 [SECURITY] #135 -- the skeleton. Workflow name,
init -> matrix -> ci-okjob graph, triggers (including the notes' "commit/open/undraft to a non-draft pr", which today's workflow does not implement -- it sets nopull_request: types:, so it uses the default[opened, synchronize, reopened]andready_for_reviewnever fires), concurrency, caching, checkout, and what stays behind inmcpx.yml. Everything else plugs into this. - 🎨 Palette: Search panel accessibility improvements #143 -- build the first real iteration and record a baseline. Split unit from e2e, put static shards in, add the lint leg, keep
nix build. (The Go cache fix it also carried landed separately in Phase 5: tag event bodies must travel with the event #182, as Phase 3: sync, blob gate, and relays #181's phase 0 asked; chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52, which the old order waited on, is closed by the same PR.) This is the issue that produces the timing data every later decision needs; without it Update dependency hyperscript.org to v0.9.93 #139 and Update dependency vitest to v5 [SECURITY] #140 are guesswork. Update dependency htmx.org to v1.9.12 #138 (in Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /sources/chat #97) is recommended just before or alongside it: it is the remaining isolation gap, and every shard inherits it. - Update dependency hyperscript.org to v0.9.93 #139 -- replace the static shards with a split driven by measured per-test time, and write down the rule for when to stop splitting. Needs 🎨 Palette: Search panel accessibility improvements #143's numbers.
t.Parallelin e2e is no longer blocked by chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52; it waits on Update dependency htmx.org to v1.9.12 #138 and a measured-parallel. - Update dependency vitest to v5 [SECURITY] #140 -- turn the recorded baseline into a gate: the budget job, and the runbook for what to do when it is breached. Needs a number, which 🎨 Palette: Search panel accessibility improvements #143 supplies.
- Bump dezren39/nix flake input to 3499ed3 — opencode now defaults to v2 #141 -- the full formatter and linter set, beyond the first lint leg 🎨 Palette: Search panel accessibility improvements #143 stands up.
- ⚡ Bolt: Optimize search match counting performance #142 --
format.yaml, last, because it only makes sense once the formatters are pinned (Bump dezren39/nix flake input to 3499ed3 — opencode now defaults to v2 #141) and there is a green test workflow for it to key on (Update Rust crate tokio to v1.38.2 [SECURITY] #135'sname:).
The notes describe an iterative loop -- split the longest job, re-measure, repeat, and do it again whenever the budget is breached. That loop is #143 -> #139 -> #140 -> #143, and it is expected to run more than once. #140's runbook is what makes it a procedure rather than a habit.
Children -- ease and impact
| # | Title | Kind | Ease | Impact |
|---|---|---|---|---|
| #135 | workflow skeleton -- init, matrix, final job, triggers, caching (cache line done in #182) | design | M | high |
| #139 | shard by measured time, and when to stop splitting | design | M | high |
| #140 | time budget in the final job, and the over-budget runbook | design + implement | S (job) + S (runbook) | med |
| #141 | every formatter and linter in the matrix, pinned by the flake | design | M (mostly clean-up, one style decision) | med |
| #142 | format.yaml -- auto-format after a green run | design | M (needs a token decision) | low-med (convenience; quality comes from the blocking check) |
| #143 | first iteration + recorded baseline | implement | M | high -- roughly halves wall clock and produces the data for everything else |
Warnings carried by this tracker
- The biggest single win is no longer blocked by chore(deps): bump build/buck/preludes/buck2-prelude from
d956689toe7457f6#52, but it is not free. 209 serial e2e tests that mostly wait on process start, sockets and runtime startup are the textbook case fort.Parallel()-- plausibly 2-3x on one runner at zero extra setup cost. Phase 5: tag event bodies must travel with the event #182 fixed the leak that made that dangerous (stop --allthrough the info files, 60 s idle exit). What still has to come first is Update dependency htmx.org to v1.9.12 #138:newEnvstill inherits the caller's environment, so an exportedMCPX_DAEMON_ENDPOINT/MCPX_SOCKETsends every parallel test to one outside daemon. And one e2e test is 3-4 processes (CLI, daemon, fake server, often a JS runtime), so-parallel 8on 4 vCPU is 24-32 processes: measure-parallel, do not take GOMAXPROCS. Update dependency hyperscript.org to v0.9.93 #139 has both. - Renaming a job invalidates every required check, and a sharded matrix produces N check names that change on every rebalance. The fix -- one
needs: [...] if: always()aggregator as the sole required check -- is in Update Rust crate tokio to v1.38.2 [SECURITY] #135, along with the trap that withoutalways()a skipped aggregator reads as neutral and a red matrix merges. - A
paths:-filtered required check never reports and blocks merges permanently. Harmless today only because nothing is required yet. - Sparse checkout breaks
nix build, which evaluates the whole flake. Sparse applies to test legs only. workflow_runonly fires from the default branch, checks out the default-branch tip rather thanhead_sha, andtypes: [completed]includes failures. All three bite ⚡ Bolt: Optimize search match counting performance #142.- The
GITHUB_TOKENloop guard is the only thing that closes the auto-format loop. Swapping in a PAT to get past branch protection re-opens it. - More shards are not monotonically better. 22-33 s of setup is the floor per leg (32.8 s on 36673523537, 22 s on 36693030957); ten legs spend 220-330 runner-seconds of setup to save ~90 s of wall clock, and the concurrency cap turns them into two waves -- which is exactly the runner-pressure effect the notes predict. Driving the fixed cost down beats adding a fifth shard.
Related
#91 (parent), #93 (#121/#122 -- the seed init mints), #97 (#134/#136 -- environment legs join this matrix), #99 (#127 -- CI is where "every test" runs so that agents do not), #49 (CI for every flake package -- overlaps this tree directly), #138 (in #97; the isolation gap before parallel shards), #178 (also edits package.nix), #52 (closed, fixed in #182), #79, #35, #65, #47, #48.
Security/auth deferred; see tracker.
Acceptance criteria
- mcpx CI runs sharded with a stated time budget; lint and auto-format checks run as their own jobs and fail the build on violations.
- The budget and shard layout are documented, and a deliberately slow test is shown to trip the budget alarm in CI.
Activity
developing-today commentedon Oct 28, 2026
CODE-828
dezren39 commentedon Oct 28, 2026
History carried over from dezren39/darwin-nix#95 (verbatim, oldest first):
@dezren39 · 2026-09-30T07:20:00Z
Integration pass. Placed in the DAG rooted at #181.
@dezren39 · 2026-09-30T12:03:33Z
Issues pass (2026-09-30, at 05c78b2).
mcpx.yml:50). The warm figure is still unmeasured (all runs since were misses by construction) -- 🎨 Palette: Search panel accessibility improvements #143 records it.checkPhase18.5 s).d956689toe7457f6#52 (closed by Phase 5: tag event bodies must travel with the event #182) no longer gates anything. The warning "chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52 blocks the biggest single win" is rewritten:t.Parallelnow waits on Update dependency htmx.org to v1.9.12 #138 (inherited environment) and a measured-parallel(3-4 processes per e2e test).@dezren39 · 2026-10-28T21:35:32Z
Triage verdict (task 2100, repo-health-nix-pilot): RELOCATED / SUPERSEDED IN PLACE — tracker stays open, but its work surface moved repos; no fix packet in this repo.
Verified against current main at
224c7b738e88c7a8e5d406987a7d88257e229db2:tracker,testing— no hard-stop labels. It tracks fast, sharded mcpx CI with a time budget, linters and auto-format, with leaves Update Rust crate tokio to v1.38.2 [SECURITY] #135, Update dependency hyperscript.org to v0.9.93 #139, Update dependency vitest to v5 [SECURITY] #140, Bump dezren39/nix flake input to 3499ed3 — opencode now defaults to v2 #141, ⚡ Bolt: Optimize search match counting performance #142 and first step 🎨 Palette: Search panel accessibility improvements #143 — all still OPEN here..github/workflows/at HEAD has nomcpx.yml. The remaining mcpx workflow,mcpx-nixbuild.yml, states in its header: "mcpx's source lives only in developing-today/code now; this repo builds it from the pinned mcpx-src flake input" (mcpx: the darwin skip override in dezren39/nix keys on this repo's check-phase line with a silent replaceStrings — no automated guard against drift #418, rewired by lab-ops-hub container crawler accepts inet_aton IPv4 spellings (127.1, 2130706433, 0x7f.1) — loopback SSRF bypass in validate_crawl_url #468) — it is a narrow build-verification check triggered only byflake.nix/flake.lock/itself..github/workflows/mcpx.ymlandmcpx-conformance.ymlat its HEAD.Route: no
small_fixpacket was enqueued in dezren39/nix — there is no in-repo CI left to shard here. The open leaves (#135, #139–#143) presuppose the removed in-repo workflow and need a relocation pass to developing-today/code (per the cross-repo rule, that move should reference this tracker rather than spawning fresh per-repo issues). #95 stays open as the nix-side tracker until that relocation is decided/done; closing it here would orphan six open leaves.dezren39 commentedon Oct 28, 2026
Intake review
🤖 Agent note · filed
Notes: Filed in the large mcpx filing batch of 2026-10-28 21:59–22:24Z and already carries a queued triage task — it is not orphaned. Verified live at intake: no merged PR references this issue and it was filed under an hour ago, so nothing can have resolved it yet. Acceptance criteria appended to the body from the issue's own stated scope; original body preserved verbatim above it.
Risks: Tracker-level scope overlaps its sibling batch issues — the triage pass must dedupe against them before any implementation packet is written, and split rather than implement if the scope is multi-PR.
Suggested approach: Triage first (task 2793): confirm the boundary against the sibling issues, then enqueue one scoped packet per deliverable slice.
24 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · triaged — task 7766 complete
52fd50d17e750ae84385b5b41414e886eccee95e; no hard-stop labels or competing PR for this tracker.Current-main verdict — tracker remains live; scope refreshed
The old single-workflow premise has changed. Current CI work is spread across
mcpx-test.yml,mcpx.yml, and the repo-wide formatter policy:mcpx-test.ymlhas unit/lint legs, six static e2e shards dealt round-robin by sorted test name, and aci-okaggregator. This is not measured-time balancing and does not implement the proposed stop rule. mcpx testing: shard the suite by measured time, and a rule for when to stop splitting #1994 remains OPEN; its existing task 8160 is currently claimed for per-test timings/LPT packing and the stop rule. No duplicate task filed.mcpx.yml(420-second default, budget report and sustained-overrun issue); mcpx testing: a CI time budget in the final job, and the over-budget runbook #1995 is CLOSED (PR mcpx: CI time budget in final job + over-budget runbook (#1995) #2708 merged). The first CI iteration is in mcpx testing: first CI iteration -- unit + e2e shards + lint + nix build, and a recorded baseline #1998 (CLOSED, PR ci(mcpx): first measured iteration — static e2e shards + unit + lint + ci-ok (#1998) #2888 merged), and the missing PR trigger from mcpx testing: CI workflow skeleton -- init, matrix, final job; triggers, concurrency, caching, checkout #1985 was added in PR ci(mcpx): add pull_request trigger to mcpx-test.yml (#1985) #5051.golangci-lintremains non-blocking until its findings are fixed; mcpx: fix the 119 golangci findings the new lint leg surfaced, then make the leg blocking #2836 is OPEN with task 8002 currently claimed. This triage did not duplicate that work.auto-format.ymland.github/ci-policy.mdintentionally make formatting non-blocking; mcpx testing: format.yaml -- auto-format after a green run; PR head vs main #1997 is CLOSED as superseded. The old requirement for a blocking formatter/format.yamlis not current policy and should not be revived.Ordering / blockers: no external blocker. The tracked remaining work is #1994/task 8160 (measured shard balancing), #2836/task 8002 (Go lint debt), then #5151/task 8832 (plugin TypeScript lint/type-check). #1901 stays open as the umbrella until the measured-sharding work and its current acceptance are reconciled.
Evidence: current workflows inspected at
52fd50d17e750ae84385b5b41414e886eccee95e; related issue and PR states checked fresh. No build/test run was applicable to this triage-only task.