Repository navigation
mcpx testing: fuzzing and property tests with one seed per run #1896
Description
Relocated from dezren39/darwin-nix#93 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).
Parent: #91
Purpose
Randomised testing for mcpx, under one rule taken from the notes and treated as non-negotiable: one seed per run generates every input. A run that fails can then be replayed exactly from a single value, and a failure that cannot be replayed is a bug in this tree rather than a fact of life.
This is the branch #91 says must land first. Every other testing tree can be built in any order and repaired later; the mapping from a seed to the data it produces cannot. Change it after seeds exist and every recorded seed silently stops reproducing what it recorded, with nothing failing to tell you. So the format, the derivation and the engine are decided here before the first randomised test is written.
Starting point, re-checked at 05c78b2: there is nothing. 735 func Test* across 64 files and not one Fuzz*, Benchmark*, testing/quick, rapid, math/rand or t.Parallel() in the tree. That is an advantage -- there is no existing randomness to migrate and no recorded seed to invalidate.
Two of these children exist because the user asked for them by name:
- chore(deps): update dependency happy-dom to v20 [security] - autoclosed #125 -- "make a simple issue for reviewing all tests, setting up framework and policies for the fuzz."
- Update dependency @tailwindcss/typography to v0.5.20 #126 -- "we aren't shooting for deterministic execution to start, maybe we should but no ... separate simple issue for considering both those things." It is explicitly not a plan; it is a record of an aspiration, kept so that the option is not lost and not mistaken for a proposal.
What this tracker does not own: when an agent is obliged to write a randomised test, and the rule that every bug gets a test -- those are policy, and belong to #99. The CI job graph that the seed's init job plugs into belongs to #95 (#135). The feature inventory those generators aim at belongs to #97.
Children
- Update Rust crate sqlx to 0.8.0 [SECURITY] #121 mcpx testing: one seed per run -- format (UUIDv7), derivation, and how it reaches every test
- Update module golang.org/x/crypto to v0.52.0 [SECURITY] #122 mcpx testing: seed store layout and collector (file per seed, artifact + scheduled collector)
- Update Rust crate tokio to v1.38.2 [SECURITY] - autoclosed #123 mcpx testing: choose the generator engine for one run-wide seed (native fuzz, testing/quick, rapid, hand-rolled)
- Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124 mcpx testing: bug-shape generators -- registry, versioning, budget, first shapes
- chore(deps): update dependency happy-dom to v20 [security] - autoclosed #125 mcpx testing: review all tests, set up the fuzz framework and its policies
- Update dependency @tailwindcss/typography to v0.5.20 #126 mcpx testing: deterministic execution as a long-term aspiration (retrofit vs Rust rewrite)
Proposed order
- Update Rust crate sqlx to 0.8.0 [SECURITY] #121 and Update Rust crate tokio to v1.38.2 [SECURITY] - autoclosed #123 together. The seed's format and derivation, and the engine that consumes it, are one decision viewed from two sides -- Go runs one test process per package, so how a single run-wide value reaches every generator depends on what the generator is. Settle both before writing any randomised test.
- chore(deps): update dependency happy-dom to v20 [security] - autoclosed #125. Audit the 735 existing tests and stand up the helper and the corpus layout the earlier two decided. This is what turns the design into something the next test can use.
- Update module golang.org/x/crypto to v0.52.0 [SECURITY] #122. The seed store and its collector. Needs Update Rust crate sqlx to 0.8.0 [SECURITY] #121's format, and needs Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95's
initjob (Update Rust crate tokio to v1.38.2 [SECURITY] #135) to exist before the artifact path is real. The collector's commits must not re-triggermcpx.yml(itspush.pathscoverspkgs/mcpx/**; see Update module golang.org/x/crypto to v0.52.0 [SECURITY] #122 step 6). - Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124. Bug-shape generators, once there is an engine and a framework to hang them on. This is the ongoing work: each escaped bug adds a generator for its shape.
- Update dependency @tailwindcss/typography to v0.5.20 #126. Any time, or never. It gates nothing.
Hard ordering constraint: nothing in #95, #97 or #99 may introduce a randomised test before #121 and #123 are decided.
Children -- ease and impact
| # | Title | Kind | Ease | Impact |
|---|---|---|---|---|
| #121 | one seed per run -- format, derivation, delivery | design | S (~150 lines helper + wrapper; ~20 lines of CI) | high -- every randomised test depends on it |
| #122 | seed store layout and collector | design | M (~80-line collector workflow, record schema, guard test) | med -- logs reproduce for 90 days without it; the store is what makes seeds durable |
| #123 | choose the generator engine | research | S (the research is the deliverable; gates #124/#125) | high -- fixes the shape of every generator |
| #124 | bug-shape generators -- registry, versioning, budget | design | M (registry + guard S; each generator S-M) | high -- the notes call this the primary testing method |
| #125 | review all tests, set up the framework and policies | review + implement | S-M (inventory S, helper S, registry + first generator S-M) | high -- turns the design into something every later test uses |
| #126 | deterministic execution / Rust -- not a plan | research | S to record; C incremental S per seam; B L-XL; A XL, not planned | low now, med long-term |
Warnings carried by this tracker
- A sixth direct Go dependency is not free.
pkgs/mcpx/package.nix:36-53opens with a comment headed "Five direct Go dependencies." that names and defends each, and states everything else is standard library. A test-only library still lands ingo.mod's direct block, changesvendorHash(package.nix:54), and becomes a Renovate bump that the existing repair step in.github/workflows/mcpx.ymlhas to service. Adoptingpgregory.net/rapidmeans editing "Five" to "Six" with a paragraph as good as the one sqlite got. Update Rust crate tokio to v1.38.2 [SECURITY] - autoclosed #123 weighs this explicitly. - Go's native fuzzing mutator is not seedable, so "one seed per run" and
go test -fuzzare not the same mechanism and cannot be made the same by wishing. Update Rust crate tokio to v1.38.2 [SECURITY] - autoclosed #123 has the detail. -shuffleinteracts with ordering, not with membership. A shuffled run and a seeded run are different kinds of reproducibility; Update Rust crate actix-files to v0.6.10 [SECURITY] #130 and Update dependency hyperscript.org to v0.9.93 #139 both touch this.- The seed variable is an undeclared
MCPX_*read. Phase 2: membership, delegation, groups, and server config #172 (in flight) is about to banos.Getenv("MCPX_*")outside the settings package except for an allowlist.MCPX_TEST_SEED(Update Rust crate sqlx to 0.8.0 [SECURITY] #121) needs an entry there, or the test helpers need to be out of that guard's scope. - Not every escaped bug has an input shape (Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124's escape loop, from its addendum): Update Rust crate axum to 0.6.20 #65 needs a hermeticity guard and fix(deps): update rust crate axum to 0.6.20 #35 an environment axis, not a generator. Classify first, so that a generator that cannot reach the bug is not counted as coverage.
Related
#91 (parent), #95 (the init job that mints the seed; #135), #97 (what the generators aim at), #99 (when an agent must write one; #129 for bugs that need a guard, not a generator), #127 (do not run the full suite to find out; since #182 fixed #52 the reason is time), #172 (seed variable vs the undeclared-MCPX_* guard), #65, #35, #74/#75, #102, #106, #108 (the escaped bugs that motivate generators).
Security/auth deferred; see tracker.
Acceptance criteria
- A fuzz/property test suite for mcpx exists in the repo, runs in CI with exactly one recorded seed per run, and a failing seed reproduces the failure deterministically.
- Coverage targets from the issue are met or explicitly re-scoped with the reason recorded on this issue.
Activity
developing-today commentedon Oct 28, 2026
CODE-823
dezren39 commentedon Oct 28, 2026
History carried over from dezren39/darwin-nix#93 (verbatim, oldest first):
@dezren39 · 2026-09-30T07:19:58Z
Integration pass. Placed in the DAG rooted at #181.
@dezren39 · 2026-09-30T12:03:38Z
Issues pass (2026-09-30, at 05c78b2).
MCPX_TEST_SEEDis an undeclaredMCPX_*read that Phase 2: membership, delegation, groups, and server config #172's coming guard must allow. Second, not every escaped bug has an input shape (Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124's classify-first step).d956689toe7457f6#52 (closed, fixed in Phase 5: tag event bodies must travel with the event #182) is replaced by Update Rust crate tracing to v0.1.40 [SECURITY] #127, because the reason not to run the full suite is now time. Added Phase 2: membership, delegation, groups, and server config #172 and Addomasnapupstream repository to flake inputs #129.@dezren39 · 2026-10-28T21:36:32Z
🤖 Agent note · triage — verdict: CONFIRMED as a tracker, but the work no longer lives in this repo — route is re-homing to developing-today/code, not a fix here
tracker+testing(no held / user/owned / do-not-merge / do-not-auto-edit). All six children are still OPEN: Update Rust crate sqlx to 0.8.0 [SECURITY] #121, Update module golang.org/x/crypto to v0.52.0 [SECURITY] #122, Update Rust crate tokio to v1.38.2 [SECURITY] - autoclosed #123, Update Rust crate tracing-subscriber to v0.3.20 [SECURITY] - autoclosed #124, chore(deps): update dependency happy-dom to v20 [security] - autoclosed #125, Update dependency @tailwindcss/typography to v0.5.20 #126 — none closed, none with a merged implementation.mainis224c7b738e88c7a8e5d406987a7d88257e229db2(laptop clone~/git/nix, fetched and fast-forwarded to origin/main). At that HEAD there is nopkgs/mcpx/tree at all: it was deleted in42b94f3"mcpx: delete the vendored copy; build from pinned developing-today/code input (fix(lab-ops-hub): judge inet_aton IPv4 spellings in the container crawl validator #469)".planning/plans/mcpx-extraction.mdrecords the extraction as done for the build switch and deletion (steps 4–5, lab-ops-hub container crawler accepts inet_aton IPv4 spellings (127.1, 2130706433, 0x7f.1) — loopback SSRF bypass in validate_crawl_url #468, 2026-10-19) and states developing-today/code is the only place the mcpx source lives; this repo now only builds it from the pinnedmcpx-srcflake input (see.github/workflows/mcpx-nixbuild.yml). So the starting point the issue describes ("735 tests, no Fuzz*/testing/quick/rapid/math/rand in pkgs/mcpx") cannot be reproduced or fixed in this repo — the tree it describes is gone.528886ebad10a4a5a0d4004cf88e0a25a1ea5258), code search finds nofunc Fuzz, notesting/quick, and noMCPX_TEST_SEED— the one-seed-per-run work has not landed there either, and no re-homed equivalent of Bump golang.org/x/crypto from 0.10.0 to 0.17.0 in /sources/graph #93 was found there.dezren39 commentedon Oct 28, 2026
Intake review
🤖 Agent note · filed
Notes: Filed in the large mcpx filing batch of 2026-10-28 21:59–22:24Z and already carries a queued triage task — it is not orphaned. Verified live at intake: no merged PR references this issue and it was filed under an hour ago, so nothing can have resolved it yet. Acceptance criteria appended to the body from the issue's own stated scope; original body preserved verbatim above it.
Risks: Tracker-level scope overlaps its sibling batch issues — the triage pass must dedupe against them before any implementation packet is written, and split rather than implement if the scope is multi-PR.
Suggested approach: Triage first (task 2790): confirm the boundary against the sibling issues, then enqueue one scoped packet per deliverable slice.
13 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 4, 2026
agent-note:v1
state: picked-up
queue: repo-health-code-pilot
task: 8719 (triage, worker repo-worker-code, lane bf419306-c4ef-470f-a553-beabf93239de)
by: openchamber-scheduled-lane
at: 2026-11-04T03:50Z
Fresh revalidation: #1896 is OPEN; no open PR matching this issue was found. The issue has legacy/conflicting status labels (
status/triagedandstatus/needs-triage) plus the queued label; the active queue task now owns the triage pass. I am checking current design/implementation state before routing.dezren39 commentedon Nov 4, 2026
Triage verdict — task 8719
Verdict: CONFIRMED live tracker; do not close. The one-seed-per-run CI acceptance remains unmet at freshly fetched
origin/maincac55070b52e14b4605b953b64c9c939705609e1:pkgs/mcpx/internal/testsupport/gen/exists, butMCPX_TEST_SEEDis absent underpkgs/mcpx. No matching open PR for #1896 was found.Current route and blockers (fresh GitHub/queue checks):
gh workflow list --allcurrently reports bothmcpx testandmcpxasdisabled_manually, so CI coverage still needs revalidation/restoration before the tracker acceptance can pass.No source/worktree changes or competing PR were created. Keep #1896 open until the seed is delivered once per run, the CI run records/replays it, and coverage targets are met or explicitly re-scoped. Existing next queue item: #1949 / task 8729.
agent-note:v1
state: triaged
queue: repo-health-code-pilot
task: 8719 (triage, worker repo-worker-code, lane bf419306-c4ef-470f-a553-beabf93239de)
by: openchamber-scheduled-lane
at: 2026-11-04T03:52Z}
dezren39 commentedon Nov 4, 2026
agent-note:v1
state: done — triaged
queue: repo-health-code-pilot
task: 8719 (triage, worker repo-worker-code, lane bf419306-c4ef-470f-a553-beabf93239de)
by: openchamber-scheduled-lane
at: 2026-11-04T03:54Z
Task 8719 completed as triaged; #1896 stays OPEN because its seed/replay-in-CI acceptance is unmet. Existing follow-up task 8729 is queued for #1949, the seed format/delivery gate. Fresh checks also show #1960 blocked behind Drew-owned PR #5127 and the
mcpx test/mcpxworkflows disabled_manually. No source or worktree changes were made.