Repository navigation
mcpx testing: policies for agents -- what to run, a test per bug, guards per bug class #1909
Description
Relocated from dezren39/darwin-nix#99 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).
Parent: #1900
Purpose
The rules agents follow when they test mcpx. Three from the notes, and one derived from the record:
- "agents should not run every single test basically ever." The reason is time. At 05c78b2 the CI test step is 148 s with
internal/e2eat 108.9 s of it (run 36693030957), up from 134.8 s / 100.7 s at d0b7e01, and every e2e test adds to it. Running it also used to leak daemons that lived for hours (chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52). Phase 5: tag event bodies must travel with the event #182 fixed that, with a leak test falsified by reverting the fix. So the rule now rests on cost alone, and the cost only grows. The one deliberate exception is the integrator's pre-merge repeat run (mcpx testing: which tests an agent runs, and never the full suite locally #1966 rule 7), which stands in for CI repeat runs until mcpx testing: flake policy -- race, shuffle, repeat runs, quarantine with expiry, a second OS #1973 exists. - "agents should make at least 1 simple, new test for every bug observed from any source, when cheap to do so -- aggresively test many permutations."
- Guard tests per bug class. The pattern already exists in the tree and works; this tracker generalises it from three or four instances into a rule.
- Flake handling, derived rather than quoted: a flake surfaced only on the lead's third full-suite run, which is the same fact as "do not run everything" seen from the other side.
The reason all of this exists, stated plainly: every serious bug in mcpx so far was found by luck.
| Found by | What it was |
|---|---|
| the lead's third full-suite run | a flake |
| CI, and only CI | a test reached the real internet (#65) |
| a Linux run | two tests that passed on macOS only (#35) |
a human reading --help |
five flags accepted and silently ignored (#74, fixed by #75) |
| a human trying it | documented capabilities that do not exist (#102, #106) |
| a human reading a flag | a wrong Node flag, accepted and inert (#108) |
an agent measuring --typecheck cost |
two concurrent runs sharing one generated program (#69); its fix broke cross-project runs (#70); the same bug one level down (#73) |
| an agent's audit | 54 of 149 settings declared and never read (#171) |
| an issue pass reading the code | a setting read but inert, which #171's brand-new guard passed (registry.pageSize, #179) |
Not one of those was caught by a test that was written to catch it. The point of this tracker is to convert each into a class that cannot recur. The last row is the sharpest: a guard written to close a class let a member of the class through, because it checks that a setting is read and not that reading it matters. #1971 owns the fix (#181 collision 15).
Guard tests are the instrument, and there are already about twenty-five of them -- not three. TestEveryDeclaredRouteAnswersOverTheSocket (internal/e2e/v1ops_test.go:287), TestEveryExecFlagChangesSomething (internal/e2e/e2e_test.go:2877, written after #74/#75), TestNoPackageRedeclaresADefault (internal/defaults/guard_test.go:92), plus route/op parity in internal/api/parity_test.go, several in internal/settings/registry_test.go, #171's internal/settings/consumed_test.go, a documentation-lists-every-setting test, TestNoOrphanProcessesAfterStop, and #182's TestEnvCleanupStopsTheDaemonItStarted. #1971 carries the corrected inventory and, more usefully, shows where the existing ones leak -- TestNoPackageRedeclaresADefault exempts seven whole packages and one file through an allowlist that nothing forces to shrink, consumed_test.go proves a setting is read but not that it matters (#181 collision 15, owned by #1971), and one guard named for reachability actually only checks name shape.
Where the policy lives is a real decision, not a formality. pkgs/mcpx/AGENTS.md is user-facing: 109 lines telling an agent how to use mcpx, loaded into every consuming agent's context, saying nothing about tests. Contributor test policy put there taxes every consumer's context window forever. #1976 owns the answer.
What this tracker does not own: the seed mechanism (#93 -- #1969 references it for the generator-per-bug-shape rule but decides nothing about format, storage or collection), the CI workflow (#95), and the integration/environment coverage (#97).
Children
- mcpx testing: which tests an agent runs, and never the full suite locally #1966 mcpx testing: which tests an agent runs, and never the full suite locally (closed)
- mcpx testing: a failing-first test for every bug, and a review of recent fixes #1969 mcpx testing: a failing-first test for every bug, and a review of recent fixes (closed)
- mcpx testing: guard tests that close a whole bug class #1971 mcpx testing: guard tests that close a whole bug class
- mcpx testing: flake policy -- race, shuffle, repeat runs, quarantine with expiry, a second OS #1973 mcpx testing: flake policy -- race, shuffle, repeat runs, quarantine with expiry, a second OS (closed)
- mcpx testing: a contributor test page agents load, kept out of the user-facing AGENTS.md #1976 mcpx testing: a contributor test page agents load, kept out of the user-facing AGENTS.md (closed)
Child links repaired 2026-11-04 (task 8016): the original children were dezren39/nix numbers 127-131, which resolve to unrelated items in this repo; the code-native children are #1966, #1969, #1971, #1973 and #1976. This tracker stays open until #1971 closes.
Proposed order
- mcpx testing: which tests an agent runs, and never the full suite locally #1966 -- the tiers, with real handles (
go test ./internal/<pkg>,-runregexes,-short, build tags, ajustrecipe) rather than prose. It is the rule agents break most often and the one that costs most when broken. Note that the rootjustfilehas no mcpx recipes at all today;pkgs/flake-tidy'stidy-testis the only per-package precedent. - mcpx testing: a contributor test page agents load, kept out of the user-facing AGENTS.md #1976 -- create the page, with mcpx testing: which tests an agent runs, and never the full suite locally #1966's tiers as its first content. Early rather than late, because a policy nothing loads is not a policy; the later issues then extend a file that already exists.
- mcpx testing: a failing-first test for every bug, and a review of recent fixes #1969 -- the failing-first rule, plus the audit of recent fixes against it. The audit is what shows whether the rule is being followed, and it is cheap now while the history is short.
- mcpx testing: guard tests that close a whole bug class #1971 -- the next guards, chosen by what the escaped-bug table demands. Exception, and it is time-sensitive: mcpx testing: guard tests that close a whole bug class #1971's settings effect guard (Phase 3: sync, blob gate, and relays #181 collision 15: a setting must be able to change an observable outcome, not merely be read) should land before or with Phase 2: membership, delegation, groups, and server config #172 (in flight), which will move up to 44 more variables into the settings registry. That one guard runs in parallel with 1-3 rather than after them.
- mcpx testing: flake policy -- race, shuffle, repeat runs, quarantine with expiry, a second OS #1973 -- flake policy. Last, because once Bump golang.org/x/crypto from 0.10.0 to 0.17.0 in /sources/graph #93's seed discipline lands a random failure is reproducible from its seed and the quarantine rules get much simpler. Until then, the integrator's pre-merge repeat runs (mcpx testing: which tests an agent runs, and never the full suite locally #1966 rule 7) are the only flake detection.
#1976 is updated as each of #1969, #1971 and #1973 lands, rather than written once.
Done since this tracker was written: #52 (closed by #182: stop --all cleanup through the info files, a 60 s idle exit in newEnv, a leak test falsified by reverting the fix); the "declared but never read" guard (#171, internal/settings/consumed_test.go). Neither was a child of this tracker. Both change what the children say, and the children have been updated.
Children -- ease and impact
| # | Title | Kind | Ease | Impact |
|---|---|---|---|---|
| #1966 | which tests an agent runs; never the full suite locally | design | S for the policy text; M with the selection script | high |
| #1969 | a failing-first test per bug, and a review of recent fixes | design + review | S (policy) / M (review + missing tests) / S (CI check) | high |
| #1971 | guard tests that close a whole bug class | design | M (policy S; each guard S-M) | high |
| #1973 | flake policy -- race, shuffle, repeat, quarantine with expiry | design | M | high |
| #1976 | a contributor test page agents load, out of the user-facing AGENTS.md | docs + implement | S | high -- it is what makes the other four reach agents |
Warnings carried by this tracker
- "Every flag passed to a subprocess is one that subprocess accepts" would not have caught Bump build/buck/preludes/buck2-prelude from
d956689tocb5dc60#108. Node 24 accepts--experimental-strip-typesand exits 0; the flag is simply inert, and--experimental-transform-typesis the one that works. An acceptance check passes. mcpx testing: guard tests that close a whole bug class #1971 writes it as a pinned capability matrix instead -- real argv against a fixture per TypeScript construct per runtime, failing when the matrix changes -- so an inert flag shows up as the Node column losing itsenumrows. The cheap exit-0 check is kept as the first half, because it still catches a typo or a runtime upgrade dropping a flag. - "Every documented capability exists" has no honest mechanism for prose. mcpx testing: guard tests that close a whole bug class #1971 rejects prose-linting outright so nobody proposes it later, and splits the ask: what has a registry behind it (Bump build/buck/preludes/buck2-prelude from
d956689tof12c8d1#102's tool-list-versus-generated-client parity, Bump h2 from 0.3.19 to 0.3.24 in /sources/pavex/examples/realworld #106's CLI-versus-/v1/execresolution) is mechanical and should not wait; prose gets a marker table, which is probably Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /sources/chat #97's feature inventory with one more column and must not be built twice. - A guard with a growing allowlist decays into a no-op.
TestNoPackageRedeclaresADefaultalready exempts seven packages and one file (guard_test.go:86-90), and nothing shrinks the list. mcpx testing: guard tests that close a whole bug class #1971's ratchet is the fix. - A regex guard that passes is indistinguishable from one that works. The defaults guard silently missed
5*time.Minutebecause gofmt writes it without spaces. Hence mcpx testing: guard tests that close a whole bug class #1971's requirement that every guard ship with a negative test proving it fails when it should. - A guard that proves "read" does not prove "works". Phase 2: delegated Manage cannot add members #171's
consumed_test.gopassedregistry.pageSize, which is read but consulted only when a caller passeslimit <= 0(internal/registry/registry.go:154-155), and no ordinary caller does (Phase 3: /id-sync/1 is used by both range and LogSync handshakes #179). mcpx testing: guard tests that close a whole bug class #1971 owns the stronger guard (Phase 3: sync, blob gate, and relays #181 collision 15). It must land before or with Phase 2: membership, delegation, groups, and server config #172. - Falsify the test, not only the fix. Update Rust crate proc-macro2 to 1.0.81 - autoclosed #75's first
--typecheckassertion matched output present with or without the flag, and was caught only because the table also asserts the without-flag case. Phase 5: tag event bodies must travel with the event #182 trusted its leak test only after reverting the fix made it fail. mcpx testing: a failing-first test for every bug, and a review of recent fixes #1969 makes this the rule. - A test that leaks a process is a failing test even when it reports pass. chore(deps): bump build/buck/preludes/buck2-prelude from
d956689toe7457f6#52 was the standing example. Phase 5: tag event bodies must travel with the event #182 fixed it and added a per-env test (TestEnvCleanupStopsTheDaemonItStarted). The class-wide backstop (a sweep afterm.Run(), plus thetestsupporttemp dirs that still leak) is a mcpx testing: guard tests that close a whole bug class #1971 candidate. chore(deps): bump build/buck/preludes/buck2-prelude fromDone in Phase 5: tag event bodies must travel with the event #182:d956689toe7457f6#52's four-hour leak is bounded today by one environment variable.autostart.idleExitis a real registered setting defaulting to4h; the e2e harness simply does not setMCPX_AUTOSTART_IDLE_EXIT. That is a one-line mitigation available now, recorded in mcpx testing: which tests an agent runs, and never the full suite locally #1966 and Update dependency htmx.org to v1.9.12 #138.newEnvsetsMCPX_AUTOSTART_IDLE_EXIT=60s(internal/e2e/e2e_test.go:68,92). Update dependency htmx.org to v1.9.12 #138 still owns the rest of the inherited environment (os.Environ(),e2e_test.go:52).- Deno's permission flags are argv that no test mentions. Searching the test files for
allow-read,allow-all,allow-envorno-checkreturns nothing (re-checked at 05c78b2) -- the same exposure as Bump build/buck/preludes/buck2-prelude fromd956689tocb5dc60#108, on a different runtime. mcpx testing: guard tests that close a whole bug class #1971's capability-matrix row covers it.
Related
#1900 (parent), #93 (#121/#124 -- the seed and the generator-per-bug-shape mechanism), #95 (#1966's "CI runs everything so agents do not" depends on CI actually doing so), #97 (#132's feature inventory is the other half of #1971's documented-capability table), #52 (closed, fixed in #182), #171 (merged: the read guard), #172 (in flight: lands with or after #1971's effect guard), #179, #181 (collision 15), #65, #35, #74 (closed, fixed in #75), #75, #78, #101, #102, #106, #108.
Security/auth deferred; see tracker.
Activity
developing-today commentedon Oct 28, 2026
CODE-835
dezren39 commentedon Oct 28, 2026
History carried over from dezren39/darwin-nix#99 (verbatim, oldest first):
@dezren39 · 2026-09-30T07:20:03Z
Integration pass. Placed in the DAG rooted at #181.
@dezren39 · 2026-09-30T12:03:40Z
Issues pass (2026-09-30, at 05c78b2).
d956689toe7457f6#52's leak. I named the integrator's pre-merge repeat run as the deliberate exception (Update Rust crate tracing to v0.1.40 [SECURITY] #127 rule 7).registry.pageSizepassing Phase 2: delegated Manage cannot add members #171's new guard (Phase 3: /id-sync/1 is used by both range and LogSync handshakes #179).omasnapupstream repository to flake inputs #129's settings effect guard (Phase 3: sync, blob gate, and relays #181 collision 15) is pulled forward. It must land before or with Phase 2: membership, delegation, groups, and server config #172, not after https://github.com/tobi/omasnap #128. Added a "done since" line for chore(deps): bump build/buck/preludes/buck2-prelude fromd956689toe7457f6#52 (Phase 5: tag event bodies must travel with the event #182) and the read guard (Phase 2: delegated Manage cannot add members #171).d956689toe7457f6#52 idle-exit stopgap is marked done by Phase 5: tag event bodies must travel with the event #182. The leak-sweep assertion now points to Addomasnapupstream repository to flake inputs #129, not Update Rust crate actix-files to v0.6.10 [SECURITY] #130, which never proposed it.outOfScopeis seven packages plus one file, not eight packages. Added warnings on read-vs-works guards and on falsifying the test (Update Rust crate proc-macro2 to 1.0.81 - autoclosed #75, Phase 5: tag event bodies must travel with the event #182).omasnapupstream repository to flake inputs #129 owns the Bump build/buck/preludes/buck2-prelude fromd956689tocb5dc60#108 capability matrix and the "documented capability exists" split, but Addomasnapupstream repository to flake inputs #129 had neither. Both are now in Addomasnapupstream repository to flake inputs #129's candidate table.@dezren39 · 2026-09-30T12:27:16Z
Issues pass (2026-09-30, architect follow-up). Related: #74 annotated "(closed, fixed in #75)".
@dezren39 · 2026-10-28T21:39:48Z
🤖 Agent note · triage — verdict: CONFIRMED as a tracker, but the work no longer lives in this repo — route is re-homing to developing-today/code, not a fix here
tracker+testing(no held / user/owned / do-not-merge / do-not-auto-edit). All five children are still OPEN: Update Rust crate tracing to v0.1.40 [SECURITY] #127, https://github.com/tobi/omasnap #128, Addomasnapupstream repository to flake inputs #129, Update Rust crate actix-files to v0.6.10 [SECURITY] #130, Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] #131 — none closed.mainis224c7b738e88c7a8e5d406987a7d88257e229db2and has nopkgs/mcpx/tree at all — it was deleted in42b94f3"mcpx: delete the vendored copy; build from pinned developing-today/code input (fix(lab-ops-hub): judge inet_aton IPv4 spellings in the container crawl validator #469)". The policy's subject file,pkgs/mcpx/AGENTS.md(whose user-facing-vs-contributor placement Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] #131 must decide), therefore no longer exists in this repo.planning/plans/mcpx-extraction.mdstates developing-today/code is the only place the mcpx source lives.3675a4703d9cbb2df858dfe5725f101cbc5dad88(laptop clone~/git/code, fetched to origin/main),pkgs/mcpx/AGENTS.mdexists (120 lines) and the guard-test pattern this tracker generalises is present and live:TestNoPackageRedeclaresADefault(internal/defaults/guard_test.go:93),TestEveryDeclaredRouteAnswersOverTheSocket(internal/e2e/v1ops_test.go:287),TestEveryExecFlagChangesSomething(internal/e2e/e2e_test.go:2957). No contributor test-policy page exists inpkgs/mcpx/docs/yet — so the tracker's substance (children Update Rust crate tracing to v0.1.40 [SECURITY] #127–Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] #131) is un-done there too, not already implemented.omasnapupstream repository to flake inputs #129's settings-effect guard time-sensitive ahead of the settings-registry move). Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99 stays OPEN here as the tracker until that re-homing happens; closing it now would lose the child list, ordering, and warnings.dezren39 commentedon Oct 29, 2026
🤖 Agent note · picked up
dezren39 commentedon Oct 29, 2026
Triage verdict — task 2804 (lane D)
Verdict: NOT a duplicate. Scope confirmed against current
pkgs/mcpxmain — the policy is still not written down anywhere an agent loads, and the gaps the body names are still live. This is a tracker; its split already exists as five relocated child issues, each open andqueue/queuedwith its own triage task. No new packet is enqueued for #1909 itself — that would duplicate the children.Dedupe
Searched the code tracker for the policies subject and each child's title. No other issue covers this tracker. It is the policies parent under strategy issue #1892 (triaged, task 2787); its guard child #1971 is also the implementation remainder named by class tracker #1893 (triaged today, task 2801) — a shared child, not a duplicate.
Children — nix numbers in the body above are historical; here is the code mapping
Proposed order from the body still stands: #1966 first (tiers with real handles), then #1976 (the page, with the tiers as first content), then #1969, then #1971 — with #1971's settings effect guard (a setting must change an observable outcome, not merely be read; the body's collision-15 item) still flagged time-sensitive — note its original reason has partly lapsed: the nix#172 move it was meant to precede was instead closed by nix PR #237 before relocation, so the guard now protects the registry as it stands rather than a pending influx. #1973 last, as the body says.
Scope confirmation vs current main
Checked at
main=8124cc0b7005(2026-10-29T05:56:32Z):TestEveryDeclaredRouteAnswersOverTheSocket(internal/e2e/v1ops_test.go),TestEveryExecFlagChangesSomething(internal/e2e/e2e_test.go),TestNoPackageRedeclaresADefault(internal/defaults/guard_test.go),TestNoOrphanProcessesAfterStop(internal/e2e/e2e_test.go), plusinternal/settings/consumed_test.goand theTestEvery*parity/dispatch family. The known leaks the body lists (hand-rowed exec-flag guard, shrink-less allowlists, read-vs-effect) are exactly mcpx testing: guard tests that close a whole bug class #1971's scope and are not yet closed.pkgs/mcpxcontains onlyAGENTS.md(the user-facing page) — no TESTING/CONTRIBUTING page. mcpx testing: a contributor test page agents load, kept out of the user-facing AGENTS.md #1976's deliverable is still owed, which is why the policy currently lives only in issue bodies like this one.justfilestill contains zero mcpx recipes (grep at this main) — mcpx testing: which tests an agent runs, and never the full suite locally #1966's "tiers with real handles, not prose" gap is unchanged since the body was written.Verification commands for the follow-ups (per the triage packet)
go test ./...inpkgs/mcpxvia.github/workflows/mcpx.ymlfor guard/policy changes; per this tracker's own first rule, follow-ups should run targeted packages (go test ./internal/<pkg>) during development and let CI run the full suite.Outcome
#1909 stays open as the policies tracker until the children land. This triage task completes; the follow-ups are the children's existing queue tasks named above.
🤖 Agent note · done
6 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · done (triaged)
Evidence (fresh, 2026-11-03)
#127–#131and parent#91are historical dezren39/nix numbers (per the body's own relocation note). In this repo they resolve to unrelated items: Update Rust crate tracing to v0.1.40 [SECURITY] #127/Addomasnapupstream repository to flake inputs #129/Update Rust crate actix-files to v0.6.10 [SECURITY] #130/Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] #131 are merged dependency-update PRs, https://github.com/tobi/omasnap #128 is a closed unrelated issue, Bump github/codeql-action from 2 to 3 #91 is a closed codeql-action bump. The re-home itself is mcpx: re-home the open mcpx backlog from dezren39/nix, with this repo as the single source #309 (CLOSED).omasnapupstream repository to flake inputs #129 (guard tests per bug class) → mcpx testing: guard tests that close a whole bug class #1971Fix path (follow-up, in order)
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 30 (priority/none). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 4, 2026
🤖 Agent note · done