mcpx testing: which tests an agent runs, and never the full suite locally #1966

Relocated from dezren39/darwin-nix#127 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #99

Summary

Decide which tests an agent working on mcpx runs, and when, so that no agent runs the whole suite as a habit. The rule: an agent runs the smallest set that covers its change -- the edited packages' unit tests, a named handful of e2e tests, and the guard tests for any registry it touched -- and leaves the full suite to CI (#95). This issue decides the tiers, how an agent finds the tests that touch a change, what is never run locally, and how a CI failure is reproduced. The page that tells agents all this is #131.

Original notes

• testing
  • importantly, as we add more testing agents should not run every single test basically ever. below documents a rough idea of a ci job design/policy, the main priority is being short as possible.

(The second line is also the joint strategy's framing in #91 and the CI tree's preamble in #95. This issue owns its agent-side half: what an agent runs locally. The CI half -- keeping the full run short -- is #95.)

Current state (re-checked at 05c78b2)

Per-package cost, CI run 36693030957 at 05c78b2 (ubuntu-latest, go test ./... -count=1, step total 148 s, the rest is compilation). The d0b7e01 run 36673523537 had e2e at 100.7 s of a 135 s step; e2e grows with every test.

Facts that shape the policy:

• e2e sees every change, through the binary, not through imports. go list -test -deps ./internal/e2e lists only internal/api and internal/testsupport. The binary under test is built by a go build ./cmd/mcpx subprocess (internal/testsupport/build.go:54-74). So Go's own dependency graph cannot say which e2e tests a change affects -- by that graph almost no change affects e2e, and in reality every change does.
• Stale green risk (not reproduced; nobody has checked it yet). go help test: results are cached for runs using only cacheable flags (-run is one), and a cached result is reused while the test binary, the env vars it reads and the files it opens are unchanged. The files the go build subprocess reads are not tracked. So after editing, say, internal/runner, go test ./internal/e2e -run '^TestX$' would plausibly print ok ... (cached) without building the new binary. CI is safe (-count=1, mcpx.yml:65). Any local e2e run must use -count=1. The same applies to a local go test ./...: its e2e line may have been cached.
• -run is an unanchored regexp (go help testflag): -run TestEveryExec also matches any longer name. Use -run '^(TestA|TestB)$'; subtests -run '^TestA$/^case$'.
• Local e2e runs no longer leak daemons (chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52, fixed in Phase 5: tag event bodies must travel with the event #182). newEnv's cleanup now runs stop --all, which reads the daemon-<key>.json info files instead of recomputing the socket path, and logs a failure instead of discarding it (internal/e2e/e2e_test.go:71-85). newEnv also sets MCPX_AUTOSTART_IDLE_EXIT=60s (:68, :92), so a daemon that escapes an interrupted run dies within a minute instead of 4h. TestEnvCleanupStopsTheDaemonItStarted (:2630) guards it and was falsified by reverting the fix. What remains: testsupport still leaves its mcpx-bin-* and mcpx-fake-* temp dirs behind on every test-binary run (build.go:27,57, os.MkdirTemp with no removal).
• Existing file split in e2e helps a little: exec_test.go, settings_test.go, v1ops_test.go, proto_test.go, consumer_test.go, runtime_test.go, logs_test.go, multiconfig_test.go, unread_test.go (Phase 2: delegated Manage cannot add members #171) -- but e2e_test.go holds 129 of the 209 tests.
• No test recipe exists anywhere: the repo-root justfile has no mcpx recipes, pkgs/mcpx has no justfile; scripts/ has only bench.sh and stress.sh (manual, real servers).
• History: Update Rust crate axum to 0.6.20 #65 was merged with its own check red and unread. Whatever the local policy, reading the PR's CI result before merging is part of it.
• In practice, integrators do run the full suite locally, repeatedly. PR bodies: Update Rust crate bacon to 2.14.0 #69 and Update Rust crate globwalk to 0.9.0 #70 "Suite 2× clean", Phase 2: delegated Manage cannot add members #171 "Suite 3× clean before my change and 2× after", Phase 5: tag event bodies must travel with the event #182 "Suite 3× clean" plus a daemon count across the runs. The merge protocol the integrating agents follow asks for this. It is the only repeat-run flake detection mcpx has until Update Rust crate actix-files to v0.6.10 [SECURITY] #130's CI jobs exist. The policy below therefore has to say who runs the full suite (the integrator, before merging), not only who does not.

Proposed policy (the tiers)

Rules around the tiers:

1. Never run go test ./... or an unfiltered go test ./internal/e2e locally, unless the user asks for it by name or you are the integrator merging (rule 7). The reason is cost: ~110 s of serial e2e plus compilation, paid again on every run, and the number grows with every e2e test. There used to be a second reason: each run left daemons alive for up to 4 h (chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52). Phase 5: tag event bodies must travel with the event #182 removed that one. The cost reason remains.
2. Always -count=1 for e2e (stale green above). For unit packages the cache is correct and welcome.
3. Push, then read CI. The full run is CI's job; an agent must read the result of the PR's check before declaring done or merging (Update Rust crate axum to 0.6.20 #65).
4. Harness changes (internal/testsupport/**, newEnv, the fake servers) affect every e2e test: run a small sample locally (^(TestNoOrphanProcessesAfterStop|TestEnvCleanupStopsTheDaemonItStarted|TestEveryDeclaredRouteAnswersOverTheSocket)$) and let CI run the rest.
5. Reproducing a CI failure: re-run only the failing test(s), by exact name, with the same run seed and shuffle order the CI shard printed (seed design and env var name: Bump golang.org/x/crypto from 0.10.0 to 0.17.0 in /sources/graph #93; shard layout: Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95). If it does not reproduce once, it is treated as a possible flake and handled by Update Rust crate actix-files to v0.6.10 [SECURITY] #130 (-count=N, -race), not by re-running the whole suite.
6. Stopgap until chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52 lands (unverified): newEnv passes os.Environ() through, and the setting's derived env name is MCPX_AUTOSTART_IDLE_EXIT (internal/settings/schema.go:225-230), so exporting e.g. MCPX_AUTOSTART_IDLE_EXIT=2m before a T2 run should bound any leak to two minutes. Check that no e2e test asserts the default first. Done in Phase 5: tag event bodies must travel with the event #182: newEnv sets MCPX_AUTOSTART_IDLE_EXIT=60s itself (internal/e2e/e2e_test.go:68,92), so there is nothing to export.
7. The integrator's pre-merge run is the exception, and it is deliberate. Whoever merges a PR runs the full suite locally more than once (current practice: 2-3 times, see Current state). Until Update Rust crate actix-files to v0.6.10 [SECURITY] #130 moves repeat runs and -race into CI, that is the only place mcpx looks for order- and timing-dependent flakes. Once Update Rust crate actix-files to v0.6.10 [SECURITY] #130's per-PR repeat job and nightly stress job exist, this step can drop to a single run or leave the local routine entirely.

Approaches: how an agent finds the tests touching a change

A -- judgment and grep (least recommended)

The agent reads its diff, greps internal/e2e for the feature's name, and picks tests. Costs nothing to set up. Risks: every agent picks differently; tests named for a behaviour rather than the flag or op (most of e2e_test.go) are missed; and an uncertain agent falls back to "run everything", which is the behaviour this issue exists to stop. Acceptable only as the documented fallback when C selects nothing.

B -- a maintained map from package/feature to tests

Split e2e_test.go by feature and keep a map (testdata/test-map.yaml or comments) from internal/<pkg> and feature names to e2e files or regexes. Deterministic and readable. Costs: the map drifts as soon as a test is added under the wrong key, splitting e2e_test.go is a large mechanical diff, and the feature inventory is owned by #97 -- a second map here would compete with it. Worth doing only if #97's inventory becomes that map (then C can read it).

C -- a selection script with a just recipe (recommended)

scripts/affected-tests.sh (or go run ./internal/testsupport/affected, to avoid bash portability questions) plus a recipe such as just mcpx-test-affected [base]:

1. git diff --name-only <base>...HEAD -- pkgs/mcpx (default base: origin/main).
2. Each edited .go file -> its package -> T1. If exported identifiers changed, add importers (T1b).
3. For e2e: pull the names the diff touches -- setting paths in internal/settings/wire.go, op names in internal/api/ops*.go, flag names in internal/cli, CLI command names, fake-server tool names -- and select e2e test functions whose bodies mention them; add the guard tests for any registry file touched (T3).
4. Print the commands and an estimate from per-test durations (from CI's go test -json artifact, Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95). Dry-run by default; --run executes them with -count=1.
5. If nothing is selected, say "no e2e selected; CI runs the full suite" -- never fall back to all.

Recommended because it gives every agent the same answer for zero context cost, it is checkable (CI can later compare its selection with the tests that actually failed, i.e. measure the miss rate), and a miss costs one CI round-trip rather than a bug, since CI still runs everything. The same selector can later feed a PR fast lane in #95 if that tree wants one.

Open questions

• Confirm the stale-cache hypothesis once, in a scratch worktree (safe to run now that Phase 5: tag event bodies must travel with the event #182 fixed chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52): edit a non-imported package, run go test ./internal/e2e -run '^TestX$' twice without -count=1, look for (cached).
• Where the recipe lives: repo-root justfile (the nix repo's) vs a new pkgs/mcpx/justfile vs only scripts/. A root recipe is discoverable; a package-local one keeps mcpx self-contained.
• The T2 budget: a count (~15 tests) or a time (~30 s)? Time is what matters; count is what an agent can see before running.
• The env var the run seed arrives in, and how a shard prints it (Bump golang.org/x/crypto from 0.10.0 to 0.17.0 in /sources/graph #93, Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95).
• Should T0 include the linters Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95 adopts (e.g. golangci-lint run ./internal/<pkg>/...)? Probably yes once they exist.

Ease · Impact · Related

Ease S for the policy text; M with the selection script (C) · Impact high
Depends on / related: #52 (closed, fixed in #182; "never locally" now rests on cost alone), #93 (run seed, reproduction), #95 (full suite, shards, go test -json timings), #97 (feature inventory, if B's map is ever built), #65 (read the check before merging), #91; siblings #131 (states this policy), #129 (T3 list), #130 (non-reproducible failures; moves the integrator's repeat runs into CI).

Security/auth deferred; see tracker.

Acceptance criteria

• The test-tier rule is decided and recorded: an agent runs the smallest set covering its change (edited packages' unit tests, a named handful of e2e tests, guard tests for any registry it touched) and never the full suite locally as a habit; the full suite is CI's job.
• How an agent finds the tests that touch a change is defined (mechanism or documented procedure).
• How a CI failure is reproduced locally is defined.
• The decisions are handed to the contributor page (mcpx testing: a contributor test page agents load, kept out of the user-facing AGENTS.md #1976) in a form it can carry verbatim.
• The work described in the Summary/Scope above is delivered via a branch + PR (no direct push to main); verification runs in GitHub Actions CI, not on the sandbox.
• Evidence (the recorded rule) is recorded on this issue before it is closed.