Repository navigation
mcpx testing: choose the generator engine for one run-wide seed (native fuzz, testing/quick, rapid, hand-rolled) #1956
Description
Relocated from dezren39/darwin-nix#123 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).
Parent: #93
Summary
Pick the engine that turns the run seed (#121) into test inputs. Candidates: Go native fuzzing (go test -fuzz), testing/quick, pgregory.net/rapid, and hand-rolled generators on math/rand/v2. The deciding constraint is the notes' hard rule -- one seed per run for all generated inputs -- plus per-test substreams and "avoid changing how seed generates data". Findings below are from docs and source; recommendation at the end, including how native fuzz corpora fit.
Original notes
- testing
- agents should make at least 1 simple, new test for every bug observed from any source, when cheap to do so-- aggresively test many permutations.
- you can absolutely rely on fuzzing anywhere you can fit it and you should whenever you can-- but you must use a single seed per-run to generate all inputs. (this could eventually be discussed with careful consideration.) it should be appended into a test tracking structured file in the repo and it can be committed. ideally main branch commit's seed or pr merge seed should make it back into file eventually. to prevent churn, maybe ci appends the seed into a gha var (race/overwrite risk?), or attached as artifact to each workflow in init job before tests begin-- a collector job can scoop them up sometime later. once a day/week/on-release/every-100-commits. im thinking touch tests/seeds/ or uv has a seed generate job. anyways ci it should end up in that dir, then as an artifact if gha and main branch, then passed into the test matrix. if uv test without arg- warn, generate and write it yourself pass it to all tests, if some other testing method there should not be a static seed, throw for empty or setup should make the seed. avoid changing how seed generates data. im not sure if a convenient uuid with time, v7?, is valid, but as long as its enough entropy whatever works. either put nothing in their or compact json, consider seeds/v1/id/<first 2 char>/<--symlink: seeds/yyyy/mm/dd/ avoiding conflicts is important.
Current state
No fuzz, property, or random tests exist (735 tests at 05c78b2, none use testing.F, testing/quick, rapid or math/rand). Go 1.26.7. Zero test-only dependencies today; go.mod deps are bubbletea/bubbles/lipgloss, yaml.v3, modernc.org/sqlite (chosen, per package.nix, to avoid a C toolchain -- the project leans toward few, pure-Go deps).
Findings
Native fuzz (testing.F) |
testing/quick |
rapid v1.3.0 | Hand-rolled on math/rand/v2 |
|
|---|---|---|---|---|
| Seedable from our run seed | No. The mutator's randomness is internal; there is no seed flag. | Yes: Config.Rand *rand.Rand (math/rand v1; wrap a ChaCha8 in a v1 Source adapter). |
Partly: -rapid.seed / RAPID_SEED (uint64, 0 = random from maphash), process-global (data.go:21 baseSeed()); settable in-process via flag.Set("rapid.seed", ...) because it is read at check time. |
Yes, fully. |
| Per-test substream keyed by test name | n/a | Yes (we build the *rand.Rand per test) |
No. Every rapid.Check in the process starts from the same base seed and uses base + iter per case (engine.go:420). Only achievable by flag.Set before each Check, which breaks under t.Parallel. |
Yes |
Runs under plain go test ./... |
Only the seed corpus (f.Add + testdata/fuzz/FuzzX/) runs, as regular tests. Actual fuzzing needs -fuzz, which must match exactly one package and one target per invocation (go help testflag). |
Yes | Yes | Yes |
| Structured inputs | Args limited to []byte, string, bool, ints, floats; structure must be decoded from bytes |
reflection over Go types; poor for "valid config document" shapes | Excellent (Custom, OneOf, SliceOfN, state machines via T.Repeat) |
Whatever we write |
| Shrinking | Yes (-fuzzminimizetime) |
No | Yes, automatic | No (cheap substitute: draw sizes small-first) |
| Coverage guidance | Yes | No | No | No |
| Output stability across upgrades | n/a | Frozen package ("not accepting new features", go doc testing/quick) |
Tied to rapid's version: fail files record rapid's internal format version (persist.go:21) and are ignored when it differs (engine.go:376) -- rapid itself treats an upgrade as possibly changing what a bitstream generates |
Ours, guarded by the golden test in #121 section 3 |
| Writes into the repo | failing inputs to testdata/fuzz/FuzzX/ (wanted: they are the regression corpus) |
no | fail files to testdata/rapid/<Test>/ unless -rapid.nofailfile |
no |
| Timing sensitivity | -fuzztime |
fixed count | cases cut short by a deadline heuristic (engine.go:413), and -short divides checks by 5 -- same seed, variable number of cases (a prefix of the same sequence) |
fixed count per budget |
| Dependency | stdlib | stdlib | 1 module, MPL-2.0, no transitive deps | stdlib |
Also: rapid's failure message prints its own -rapid.seed=N, not our run seed, so the reproduce line would need translating; and rapid.MakeFuzz(prop) turns a rapid property into a native fuzz target (bytes become rapid's bitstream).
Approaches
A -- testing/quick (least recommended)
Seedable and stdlib, but frozen, reflection-driven (hard to express "a valid layered config" or "a JSON Schema"), no shrinking, and on math/rand v1. Nothing it offers is not better in B or C.
B -- rapid as the primary engine
The best off-the-shelf generator and shrinker in Go; property code is pleasant (rapid.Check(t, func(t *rapid.T){ ... Draw(t, "x") })), and MakeFuzz bridges to native fuzzing for free. Seeding works via RAPID_SEED derived from the run seed (#121 section 2), so the "one seed per run" rule is satisfiable. The costs: no per-test substream (all tests share base + iter; adding a case to one property does not shift another, but the scheme is rapid's, not ours), generation output is coupled to rapid's version (an upgrade is an implicit generator-version bump for every rapid-backed generator -- in tension with "avoid changing how seed generates data"), its deadline heuristic makes the case count timing-dependent, and it writes fail files into testdata/rapid/ unless disabled. It would also be mcpx's sixth direct Go dependency. package.nix:36-53 names and defends each of the five, and a test-only module still lands in go.mod's direct block and changes vendorHash (package.nix:54). Adopting rapid therefore means amending that comment in the same PR (addendum below). A defensible choice if shrinking turns out to matter more than stream control.
C -- hand-rolled generators on ChaCha8 behind a tiny Source interface, native fuzz for parsers (recommended)
internal/testsupport/gen: generators arefunc(src Source, size int) TwhereSourceis justUint64() uint64. In property mode,srcis ChaCha8 keyed bystream(t, gen@version)from Update Rust crate sqlx to 0.8.0 [SECURITY] #121.- The same generator also serves native fuzzing: a
bytesSourcethat readsUint64s from the fuzzer's[]byte(zeros when exhausted) lets any generator become aFuzz*target -- the patternrapid.MakeFuzzuses, in ~20 lines. So coverage-guided fuzzing of structured shapes is available without a second generator. - Native
Fuzz*targets for byte-oriented parsers where coverage guidance pays:config.StripJSONC(internal/config/config.go:718),codegen.TypeForover raw schema bytes (internal/codegen/ts.go:70),launcher.Expandtemplates (internal/launcher/template.go:80),settings.ParseBytes(internal/settings/value.go:415). Their seed corpora (f.Add+testdata/fuzz/FuzzX/) run in everygo testas deterministic regression tests; the minimized failing input from a seeded failure is committed there too. Long-fuzzsessions are a scheduled/soak job (one target per invocation) owned by Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95's budget, not part of the per-run seed rule -- they are explicitly outside "one seed per run", which the notes' "(this could eventually be discussed with careful consideration.)" leaves room for. - Shrinking substitute: generators take a
sizethat the runner grows from small to large across a test's budget, so the first failure is usually small; revisit rapid if that is not enough. - Why C: it is the only option that satisfies all three hard requirements (one run seed, per-test substreams, versioned stable output) without fighting the library; it is stdlib-only; and it keeps a clean upgrade path to B (rapid can later be driven by
RAPID_SEEDfrom the same root).
Open questions
- Is shrinking worth a dependency? Decide after the first ~5 generators exist (chore(deps): update dependency happy-dom to v20 [security] - autoclosed #125): if failures are routinely unreadable, adopt rapid for new generators (B) with its version pinned and treated as a generator-version input.
- Native fuzz corpora: commit only failure-derived inputs, or also a curated set of generator outputs (
gen -> testdata/fuzz/FuzzX/<hash>)? - Scheduled
-fuzzsoak: which targets, how long, on which runner (Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95)? It cannot share the run seed (the mutator is unseeded); is that acceptable as a documented exception? - Does
testing/synctest(Go 1.25+) belong in the same helper package for time-dependent properties (Update dependency @tailwindcss/typography to v0.5.20 #126)?
Ease · Impact · Related
Ease S (research is done here; the choice gates #124/#125) · Impact high (fixes the shape of every generator)
Depends on / related: #121 (seed, derivation, golden guard), #124 (generator registry), #125 (first generator), #126; #95 (soak -fuzz job and budget), #99 (policy on committing fuzz corpora).
Security/auth deferred; see tracker.
Activity
developing-today commentedon Oct 28, 2026
CODE-881
dezren39 commentedon Oct 28, 2026
History carried over from dezren39/darwin-nix#123 (verbatim, oldest first):
@dezren39 · 2026-09-30T06:44:04Z
Addendum from a second pass over the same slice (a duplicate fuzz worker ran concurrently; this issue got there first and is the one to keep). Three facts that are not in the body and that strengthen its recommendation C.
rapid would be mcpx's sixth direct Go dependency, and this repo has a written policy about that
The Findings table costs rapid as "1 module, MPL-2.0, no transitive deps". That undercounts it.
pkgs/mcpx/package.nix:36-53opens with a comment block headed:and then spends seventeen lines defending each one individually —
modernc.org/sqliteis the pure-Go translation specifically so the derivation needs no C toolchain and cross-compilation still works; bubbletea/bubbles/lipgloss are costed at 1.2 MB in the stripped binary and justified as replacing terminal-handling code "that is never finished and never correct on every terminal"; yaml.v3 exists because OpenAPI documents are published as YAML as often as JSON. It closes:go.modmatches: exactly five entries in the directrequireblock."Test-only dependency" is not an escape hatch from that policy:
go.mod's direct require block, not the indirect one. The comment's count becomes wrong the day it is added.vendorHash(package.nix:54)..github/workflows/mcpx.ymlexists precisely because dependency bumps break the nix build and merged silently until someone rebuilt.So adopting rapid means editing that comment block from "Five" to "Six" and writing rapid a paragraph as good as sqlite's. That is a fair price for a genuine need; it is not free, and it should be stated in Approach B rather than discovered in review.
None of this changes the recommendation — it makes C stronger, and it sharpens the Open question "Is shrinking worth a dependency?" into a concrete deliverable: if rapid is adopted, the PR must also amend
package.nix:36-53. Worth adding that as an acceptance criterion on whichever issue eventually adopts it.F.Fuzz's accepted argument types, for the recordThe Findings table says "args limited to
[]byte,string, bool, ints, floats". The exact list, since it bounds which parsers can be native fuzz targets without a byte-decoder:This is why the
bytesSourcebridge in Approach C is load-bearing rather than a convenience: it is the only way a structured generator reaches coverage-guided fuzzing without a second generator.The free half of native fuzzing, stated explicitly
Approach C mentions it; it is worth making unmissable because it is the cheapest thing in this tracker. With
-fuzzunset, aFuzz*target still runs against its seed corpus as an ordinary test —F.Addentries plus every file intestdata/fuzz/<Name>/(go doc testing, lines 261-264). Andgo test -fuzzwrites failing inputs into that same directory automatically (lines 254-258).So a committed corpus is a regression suite that runs on every CI run with zero bookkeeping, and the soak job files its own regressions. That is #99's "one test per bug" delivered mechanically for the byte-shaped targets, which is a much stronger argument for native fuzz targets than coverage guidance alone.
@dezren39 · 2026-09-30T07:20:50Z
Integration pass. Placed in the DAG rooted at #181.
@dezren39 · 2026-09-30T12:03:24Z
Issues pass (2026-09-30, at 05c78b2).
testing.F,testing/quick, rapid ormath/rand. Re-citedsettings.ParseBytestointernal/settings/value.go:415(it was :375).package.nix:36-53, and it changesvendorHash(:54). The addendum asked for this to be "stated in Approach B rather than discovered in review".dezren39 commentedon Oct 29, 2026
🤖 Agent note · handed off (input from #1896 triage, task 2790)
One fact for this issue's engine decision, found while triaging the parent tracker #1896 against current main (
8124cc0b7005, 2026-10-29):The "sixth direct dependency" framing in #1896's body is stale.
pkgs/mcpx/package.nix(lines 36–53) still opens with "Five direct Go dependencies" and narrates five (modernc.org/sqlite, bubbletea, bubbles, lipgloss, yaml.v3), butpkgs/mcpx/go.modnow lists seven direct requires — the five plusgithub.com/santhosh-tekuri/jsonschema/v6andgithub.com/tetratelabs/wazero. So adoptingpgregory.net/rapidwould be direct dependency eight, and the package.nix comment needs a recount paragraph whichever engine is chosen. Verification for any dependency change runs in CI (go test ./...inpkgs/mcpxviamcpx.yml+ the nix build, which pinsvendorHash).Full triage verdict: #1896 (triage comment by repo-worker-code, task 2790).
dezren39 commentedon Oct 29, 2026
🤖 Agent note · picked up
dezren39 commentedon Oct 29, 2026
🤖 Agent note · done
Triage verdict: CONFIRMED — recommendation C stands; implementation follow-up enqueued
Dedupe (checked 2026-10-29): No duplicate. Searches for "generator engine", "rapid", and "testsupport gen ChaCha8" surface no other issue covering the engine choice or a
testsupport/genpackage. #1958 (bug-shape generator registry) consumes this engine; #1949 supplies the seed/derivation it keys from; #1896 is the parent. This issue is the sole owner of the engine decision.Scope confirmed against current
pkgs/mcpxmain (fresh checks 2026-10-29):go.modcarries exactly the five direct deps the issue names (bubbles, bubbletea, lipgloss, yaml.v3, modernc.org/sqlite) — the "rapid would be a sixth direct dep +package.nixamendment" cost is real.internal/testsupportdoes not exist (404); code search finds notesting.Fand no generator use ofmath/randinpkgs/mcpx. The onlymath/rand/v2uses aretime.Sleep(rand.N(40ms))jitter atinternal/e2e/keepalive_takeover_test.go:253andinternal/daemon/handoff_conns_test.go:289— not input generation, so the issue's "no fuzz/property/random generation" claim holds in substance.internal/config/config.go,internal/codegen/ts.go,internal/launcher/template.go,internal/settings/value.go.Decision carried forward: Approach C (hand-rolled generators on ChaCha8 behind a
Sourceinterface + native fuzz for parsers) is the only candidate meeting all three hard requirements — one run seed, per-test substreams, versioned stable output — and it is stdlib-only. The follow-up (task 3251) is deliberately scoped to the core:internal/testsupport/gen(Source, ChaCha8 stream keyed per #1949 derivation,bytesSourcefuzz adapter, size-growing runner), the golden stability guard, and ONE proof target (FuzzStripJSONC). The remaining three parser targets, the registry (#1958), and the-fuzzsoak job stay as named follow-on work on this issue; rapid adoption stays deferred per the issue's own open question (revisit after ~5 generators exist).This issue stays open as the tracking issue for task 3251 and the follow-on targets.
23 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · done (triage verdict)
Triage verdict: DECISION LANDED — close; residuals owned by #1949, #1985, #1960, #1896
Checked against
origin/main1e3763faf4e3(2026-11-03) with fresh reads, no cached state:c63e5af54796033dc25e236be391fea0a100e936) is an ancestor of main, andpkgs/mcpx/internal/testsupport/gen/exists on main:gen.go,generators.go,registry.go, plusgen_test.goandregistry_test.go.gen.go's package doc names Approach C and states the derivation. feat(mcpx): testsupport/gen core (ChaCha8 streams) + FuzzStripJSONC proof target (#1956) #2636 touched onlyinternal/testsupport/gen/{gen.go,gen_test.go}and the proof target — nogo.mod/go.sumchange, i.e. the choice cost zero new direct dependencies, as Approach C claimed.Root()=SHA-256("mcpx-test-seed/v1\x00" || seedID);StreamKey()=SHA-256(root || 0x00 || "g@v" || 0x00 || testName)→ ChaCha8 key;Stream/ForTesthand out per-test substreams. So "one run-wide seed, per-test substreams keyed by test name, version-stable output" holds by construction, not by convention.registry.go/registry_test.goare on main.internal/config/stripjsonc_fuzz_test.go:19(FuzzStripJSONC, drawing throughtestsupport/gen) with five committed seeds underinternal/config/testdata/fuzz/FuzzStripJSONC/; the corpus runs as ordinary tests under plaingo test ./....grep "func Fuzz" pkgs/mcpxreturns onlyFuzzStripJSONC. Targets:codegen.TypeFor(internal/codegen/ts.go:88),launcher.Expand(internal/launcher/template.go:82),settings.ParseBytes(internal/settings/value.go:457). They have no issue of their own; mcpx testing: review all tests, set up the fuzz framework and its policies #1960 (fuzz framework and policies, open) is their natural owner, and I recorded them there.Correction to the dependency-policy framing (this body and the 2026-10-29 note)
pkgs/mcpx/go.modtoday has eleven direct requires — bubbles, bubbletea, lipgloss,mattn/go-runewidth,santhosh-tekuri/jsonschema/v6,tetratelabs/wazero,golang.org/x/net,golang.org/x/sys,yaml.v3,modernc.org/sqlite,tailscale.com— on Go 1.27.2. So rapid would have been direct dependency #12, not #6/#7/#8; the "sixth dependency" cost argument is stale in the same direction it has been stale twice already.pkgs/mcpx/package.nix:50still opens# Five direct Go dependencies.— a standing doc defect independent of this decision (whichever PR next touchespackage.nixshould recount, or the count should be derived). It did not need fixing here: Approach C shipped stdlib-only.Disposition
Closing as completed: the question this issue asks — which engine turns the run seed into test inputs — is answered (Approach C: hand-rolled generators on ChaCha8 behind a
Sourceinterface, stdlib only, nativetesting.Ffor byte-oriented parsers) and the answer is merged. Keeping it open only re-feeds coverage passes, which is why task 7804 exists.Residual work, by owner: #1949 (seed id/plumbing) · #1985 (CI workflow skeleton, soak
-fuzzbudget) · #1960 (fuzz framework, policies, the three remaining targets) · #1896 (parent tracker).dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 30 (priority/none). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03