mcpx testing: a labelled real-software subset, pinned by the flake, run nightly #1990

Relocated from dezren39/darwin-nix#137 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #97

Summary

Define and label the subset of integration tests that must work end to end with real software: real MCP servers, real runtimes, and real network where the server needs it. Pin the versions they run against in one place, the flake. #134 recommends building the test image from the flake so there is one version set; #182's user-facing pkgs/mcpx/Containerfile pins its own (Go via golang:1.26-bookworm, bun via oven/bun:1-debian), so today there are two, and they already move separately (Renovate #184). The protocol half of the pin is the binary's own report, stamped into both (below). Run the subset nightly and on release, not per PR, because real servers bring network access and flakiness (#65). Fold the manual scripts/stress.sh into the subset.

Original notes

• testing
  • a general goal is to ensure all features have one or more integration tests
    • the mcp servers can be specialized mocks but the app entrypoint or binary is used and config is passed in or config path set to appropriate test
    • for both binary && app entrypoint (consider binary a superset)
      • all tests should pass for nix flake & container
        • a specific subset should completely work with all real software
          • these should be found/labeled whenever possible
          • the supported mcp version is the one in the flake/containerfile

Current state (main d0b7e01)

• All e2e tests use fakes (199 at d0b7e01, 209 at 05c78b2). fakemcp/askmcp (internal/testsupport/) and a stub registry (MCPX_REGISTRY_URL, internal/e2e/e2e_test.go:52-57, since Update Rust crate axum to 0.6.20 #65). Nothing in CI runs a real MCP server.

• The runtimes are real but unpinned in CI. mcpx.yml installs setup-deno v2.x, setup-bun latest and node 24. package.nix wraps the nixpkgs versions instead. The runtime version CI tests is not the one users of the package get (Update Rust crate rand to v0.9.3 [SECURITY] #134).

• scripts/stress.sh checks discovery latency under load (<200 ms per mcpx ls after 30 concurrent), concurrent isolation on a stateful server (tools.chrome_devtools.new_page/list_pages, lines 36-40), and leaks. It needs MCPX=<binary> and runs against whatever .mcpx.json the caller's config chain resolves, so the servers and their versions are the user's, not pinned. It is manual, and so is scripts/bench.sh. It uses python3 for arithmetic.

• The MCP protocol version lives in Go source, not the flake:

  • the client speaks 2025-11-25 (internal/mcpclient/client.go:22);
  • the server side serves 2025-03-26 (internal/mcpserver/revisions.go:14, Oldest) up to the newest, with per-feature floors (revisions.go:85-87, conn.go:192,198);
  • the full served list is Supported = {"2026-07-28", "2025-11-25", "2025-06-18", "2025-03-26"} (internal/mcpserver/server.go:778);
  • the mocks already span two revisions: fakemcp announces 2025-06-18 (internal/testsupport/fakemcp/main.go:112), askmcp 2025-11-25 (askmcp/main.go:145) -- keep that; it gives the negotiation some coverage for free (from Phase 5: alias display rules #164);
  • the protocol matrix is served at /v1/protocol and tested by TestTheProtocolMatrixIsServedFromTheTableTheCodeReads (internal/e2e/proto_test.go:567).

  So "the supported mcp version" has two parts. The protocol revisions are fixed by the mcpx source at a commit. The real server implementations and their versions are what the flake can pin.

  How to pin the first part, from Phase 5: alias display rules #164: there is no single "supported mcp version" to type into the flake or a Containerfile -- it is a list plus a client constant, and /v1/protocol already reports both. So (1) in the gating mock suite, assert /v1/protocol against a checked-in golden file, so any change to Supported or mcpclient.ProtocolVersion is a reviewed diff; (2) stamp the same list onto the artifacts generated from the binary at build time, the way postInstall generates the man page -- e.g. a passthru attribute on the nix package and an OCI label on the image -- never transcribed by hand; (3) in the real suite, assert each pinned server negotiated a revision in Supported and record which. The image label would live in Phase 5: tag event bodies must travel with the event #182's Containerfile, which the GIT architect owns; this is input for Update Rust crate rand to v0.9.3 [SECURITY] #134, not a design of that file. The golden file is the same "generated reference that cannot drift" problem as Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/import #96/⚡ Bolt: Optimize cursor color generation and position grouping #145 -- one mechanism for all of them.

• Real servers available in the pinned nixpkgs (c7def04, pkgs/by-name):

  • the reference servers mcp-server-fetch 2026.8.31, mcp-server-filesystem 2026.8.18, mcp-server-git 2026.8.31, mcp-server-memory 2026.8.18, mcp-server-time 2026.8.31 and mcp-server-sequential-thinking;
  • playwright-mcp 0.0.76, github-mcp-server 1.11.0, context7-mcp 4.0.3, firefox-devtools-mcp, mcp-proxy, and others.

  Not in nixpkgs: @modelcontextprotocol/server-everything, the reference server that exercises every protocol feature (prompts, resources, templates, sampling, elicitation, notifications, long-running progress). Also absent: chrome-devtools-mcp, which stress.sh uses.

What the subset must cover

"completely work with all real software" is too broad to be a test, so here it becomes: every feature in the #132 inventory, exercised once against a real server that implements it, plus the runtimes at pinned versions.

Approaches -- how the subset is labelled

A -- a test-name prefix, TestReal... (least recommended)

Selectable with go test -run '^TestReal', and visible in output. Nothing stops a real-server test from going unnamed, and nothing stops a TestReal test from quietly using the fake. On a machine without the server, the test must skip itself, and the skip is invisible. That is the #35 lesson again. (testing.Short() with CI passing -short is worse still: it is opt-out, so the obvious go test ./... on a laptop runs the real servers -- the property that produced #65 -- and -short already means "skip slow", not "skip network"; from #164.)

B -- a manifest file

internal/e2e/realsoftware.json lists test names with the server and version each needs. A guard fails if a listed test does not exist or an unlisted test starts a non-fake server. It is explicit and reviewable, and gives a place to record versions. It is also a second list to keep in sync, and the guard needs to know what counts as "a non-fake server". #164's variant lists servers instead of tests: a checked-in .mcpx.json with every server at an exact version (npx -y @modelcontextprotocol/server-filesystem@1.2.3, never @latest), bumped by Renovate as a reviewable diff. Simpler than a nix derivation, but it needs the network at test time; C's flake-built servers do not.

C -- a build tag realsoftware plus a harness helper (recommended)

• The real-software tests live in internal/e2e/real_*_test.go behind //go:build realsoftware. They do not compile into the per-PR run, so they cannot slow it down or flake it.
• Each test calls testsupport.RealServer(t, "mcp-server-time"), which resolves the server from MCPX_REAL_SERVERS_DIR (a nix-built directory of pinned servers: nix build .#mcpx-real-servers). Under MCPX_TEST_REQUIRE (Update Rust crate rand to v0.9.3 [SECURITY] #134) a missing server fails; without it, it skips with the name.
• RealServer records the name and version into the Update dependency @chenglou/pretext to ^0.0.9 [SECURITY] #132 ledger. The ledger then serves as the "found/labeled" list, generated, not maintained by hand.
• "these should be found/labeled whenever possible": a guard in the default build greps real_*_test.go for any test that does not call RealServer (it would be a fake test hiding in the tag), and the default-build tests for RealServer calls outside the tag.

Recommended because the tag keeps the per-PR budget clean (#95), and the helper makes the label a fact about what the test does, not a naming convention. Pinning comes from the flake, so "the supported version" is nix flake metadata plus the mcpx-real-servers derivation.

A compile-time tag has a cost: the tagged files are type-checked only when the tag is built. So the nightly job must run go vet -tags realsoftware ./internal/e2e, and it is cheap to also do so in tier 1.

Where it runs

• Nightly and on release (tier 2, Update dependency happy-dom to v21 [SECURITY] #136), in the flake image (Update Rust crate rand to v0.9.3 [SECURITY] #134) with mcpx-real-servers added. No internet except the registry job.

• stress.sh folded in:

  • its latency check becomes a realsoftware test with a generous threshold (the 200 ms figure is a laptop number, so record it and alarm on regression rather than failing on an absolute value);
  • its isolation rounds become a test using playwright-mcp;
  • its leak check becomes an assertion on Update dependency htmx.org to v1.9.12 #138's isolated state dir.

  Keep the script as a thin wrapper for manual runs against your own config, which is still useful and is the only place "the user's servers" belong.

• On a flake update: because the servers are pinned, a nix flake update that bumps a server is exactly when the subset must run. Trigger it on flake.lock changes (Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95's path filters).

• Flakiness policy (Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99): a real-software failure opens an issue automatically and does not block. A test that is flaky twice is quarantined with an issue, not retried silently.

Open questions

• Package server-everything in pkgs/ (it is an npm package, so it needs an npmDeps hash), or run it via npx at a pinned version? The first is hermetic; the second needs the network.
• Is there an official MCP conformance suite worth running against mcpx serve//mcp as a server? mcpx is also an MCP server, and that direction has no real-client test today. Real clients to consider are opencode itself and the MCP Inspector CLI.
• Which protocol revisions must the subset cover: only the client's 2025-11-25, or also old-revision servers (2025-03-26, 2025-06-18), since mcpx negotiates down? A pinned old version of a reference server would cover the latter.
• Should the pinned server versions be printed by mcpx doctor or /v1/protocol in the test image, so a bug report says which ones were tested?
• A real-software test that randomises (for example, concurrency order) takes the run seed (Bump golang.org/x/crypto from 0.10.0 to 0.17.0 in /sources/graph #93).
• Does a feature tested only against real software count as covered in Update dependency @chenglou/pretext to ^0.0.9 [SECURITY] #132's ledger? Phase 5: alias display rules #164 argued it must be excluded from the denominator: "only tested against real software" is a finding, not coverage, and counting it hides gaps behind a nightly job. RealServer can still record into the ledger as a label (C above) without counting as coverage.
• A pinned server withdrawn upstream: fail, or report unavailable and continue? Probably fail on a pre-release run and report on a nightly (from Phase 5: alias display rules #164).
• Supported includes 2026-07-28: does any real server speak it yet, and can the subset prove mcpx's newest-first negotiation downgrades correctly against servers that do not (from Phase 5: alias display rules #164)?
• An HTTP/SSE-transport server belongs in the set, since docs/story.md:790 promises "Streamable HTTP and SSE are spoken directly; no mcp-remote" (from Phase 5: alias display rules #164; mcp-proxy in the table above is one way).
• Run the subset in the flake image (Update Rust crate rand to v0.9.3 [SECURITY] #134) with a warmed npm/uv cache? It is the difference between a short nightly and a long one if any server is still fetched (from Phase 5: alias display rules #164).

Ease · Impact · Related

Ease M (tag, helper, reference servers: M; server-everything packaging: S-M; stress.sh fold: S) · Impact med-high (the only proof mcpx works with anything but its own fakes)
Depends on / related: #132 (ledger, inventory), #134 (flake image, MCPX_TEST_REQUIRE), #138 (isolation), #136 (tiers), #65 (no internet in normal tests; registry isolated), #95 (nightly/release workflow, flake.lock trigger), #99 (flake quarantine policy), #93 (seed), #49 (building mcpx-real-servers as a flake package).

Security/auth deferred; see tracker.

Acceptance criteria

• A labelled real-software subset, pinned by the flake, runs nightly in CI and its results are visible/recorded as the body specifies.
• The work described in the Summary/Scope above is delivered via a branch + PR (no direct push to main); verification runs in GitHub Actions CI, not on the sandbox.
• Evidence (CI links, timings, or probe output as applicable) is recorded on this issue before it is closed.