mcpx testing: review all tests, set up the fuzz framework and its policies #1960

Relocated from dezren39/darwin-nix#125 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #93

Summary

The simple, explicitly requested issue: review all existing tests, set up the fuzz framework, and write its policies -- i.e. execute #121-#124 in the smallest useful slice. Three deliverables: an inventory of the 735 tests (at 05c78b2; which become generators, which hide time/randomness/ports, which are flake-prone), the seed helper + wrapper + first generator landed, and the policy text handed to #99.

Original notes

• testing
  • agents should make at least 1 simple, new test for every bug observed from any source, when cheap to do so-- aggresively test many permutations.
    • you can absolutely rely on fuzzing anywhere you can fit it and you should whenever you can-- but you must use a single seed per-run to generate all inputs. (this could eventually be discussed with careful consideration.) it should be appended into a test tracking structured file in the repo and it can be committed. ideally main branch commit's seed or pr merge seed should make it back into file eventually. to prevent churn, maybe ci appends the seed into a gha var (race/overwrite risk?), or attached as artifact to each workflow in init job before tests begin-- a collector job can scoop them up sometime later. once a day/week/on-release/every-100-commits. im thinking touch tests/seeds/ or uv has a seed generate job. anyways ci it should end up in that dir, then as an artifact if gha and main branch, then passed into the test matrix. if uv test without arg- warn, generate and write it yourself pass it to all tests, if some other testing method there should not be a static seed, throw for empty or setup should make the seed. avoid changing how seed generates data. im not sure if a convenient uuid with time, v7?, is valid, but as long as its enough entropy whatever works. either put nothing in their or compact json, consider seeds/v1/id/<first 2 char>/<--symlink: seeds/yyyy/mm/dd/ avoiding conflicts is important.
      • think a hack job of antithesis/tigerbeetle. this is the primary testing method is throw insane amount of variations. if something is found outside this system and a test that could have had missed it, make a different generator that uses the same seed and makes an infinite amount of things matching the offensive shape.
        • make a simple issue for reviewing all tests, setting up framework and policies for the fuzz.

Current state (initial inventory from rg, re-counted at 05c78b2 -- the review deepens it)

Totals: 9 test files call time.Sleep, 14 call time.Now; at least 9 files use []struct tables (37 match a looser table/range pattern); no t.Parallel() anywhere; network use is unix sockets and httptest (daemon/routes_resolve_test.go, openapi, registry), plus the stub registry in e2e after #65.

Plan (small steps; each a PR)

1. Inventory: finish the table above for every package as a checklist in this issue (or docs/testing/inventory.md if Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99 picks a docs home). Per test file: generator candidate / hidden time / hidden randomness (production code mints ids with crypto/rand in pool, tasks, elicit, artifacts, mcpserver, logging/trace, so outputs differ per run; flag any test that asserts on them) / flake-prone (sleep-then-assert) / fine. No rewrites in this step.
2. Seed helper + wrapper (Update Rust crate sqlx to 0.8.0 [SECURITY] #121 option C): internal/testsupport/seed (Rand(t, gen), failure reproduce line, CI fail-closed), the wrapper recipe, .gitignore for .cache/test-seed/. Unit-test the helper itself (derivation is stable: golden root/stream for a test-vector id).
3. Registry + first generator (Update Rust crate tokio to v1.38.2 [SECURITY] - autoclosed #123 option C, Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124): internal/testsupport/gen with Source, bytesSource, golden-output guard, and one generator. Proposed first: launcher.template or codegen.jsonschema -- pure functions, fast, rich shapes. Plus its Fuzz* twin with a small seed corpus.
4. CI init job (Update Rust crate sqlx to 0.8.0 [SECURITY] #121 section 7): mint + artifact + MCPX_TEST_SEED on the test step; coordinate the edit with Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95 (who owns the workflow).
5. Policy text handed to Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99 (which owns where it lives and the per-bug-test policy): the rules below.
6. Fix the two cheapest flake-prone spots found in step 1 (e.g. pool_test.go:423 with a fixed now), as proof the inventory pays.

Deferred to follow-ups: remaining generators from #124, the collector (#122), soak fuzzing (#95).

Proposed policy text (for #99 to place)

• Every random input in a test comes from seed.Rand(t, ...). No math/rand top-level functions, no time.Now().UnixNano() seeds, no crypto/rand for test data. (A guard test can rg for these in _test.go.)
• Never hard-code a run seed. Test vectors inside generator golden tests are the only fixed keys.
• Run tests through the wrapper; in CI a missing seed is a failure.
• A failing seeded test is reported with its seed and reproduce command; do not re-run to green without recording the seed (Update module golang.org/x/crypto to v0.52.0 [SECURITY] #122 keeps failing seeds).
• Changing a generator's output means bumping its version.
• When a bug escapes, add the regression test and answer "which generator could have caught this?" (Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124 escape loop).

Approaches

No real three-way choice: the user asked for one simple issue that executes the designs. The one judgement call is order -- inventory first (read-only, cheap, informs which generator goes first) vs helper first (unblocks everyone). Proposed: helper and inventory in parallel, since neither depends on the other; first generator after both.

Open questions

• Inventory as an issue checklist (cheap, dies with the issue) or a committed doc (maintained, discoverable)? Defer to Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99's decision on where testing docs live.
• First generator: launcher.template (smallest) or codegen.jsonschema (highest yield, needs one deno check)?
• Guard test banning unseeded randomness in _test.go: now, or once the first generators exist?

Ease · Impact · Related

Ease S-M (inventory S, helper S, registry + first generator S-M) · Impact high (turns the design into something every later test uses)
Depends on / related: #121, #122, #123, #124, #126; #99 (policy home, per-bug test, flake policy), #95 (CI init job), #52 (closed, fixed in #182; the leftover temp dirs are a #129 guard candidate), #65, #35.

Security/auth deferred; see tracker.

Acceptance criteria

• An inventory of the existing tests (735 at 05c78b2) is recorded: which become generators, which hide time/randomness/ports, and which are flake-prone.
• The fuzz framework's first slice is landed via PR: the seed helper, the wrapper, and the first generator.
• The fuzz policy text is written and handed to Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99 (linked here).
• The work described in the Summary/Scope above is delivered via a branch + PR (no direct push to main); verification runs in GitHub Actions CI, not on the sandbox.
• Evidence (CI links and the inventory location) is recorded on this issue before it is closed.