mcpx testing: an in-process app entrypoint beside the binary suite, and a scriptable mock MCP server #1980

Relocated from dezren39/darwin-nix#133 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #97

Summary

Today every e2e test runs mcpx as a subprocess. This issue proposes an in-process entrypoint alongside it: move the dispatch in cmd/mcpx/main.go into cli.Run(ctx, args, env, stdio) int. Then run most integration tests in-process, which is faster, gives coverage, and lets you set breakpoints. Keep a binary suite as a superset that also covers what only a real process has: signals, exit codes, argv parsing, the detached daemon, and the nix wrapper. Let the binary suite point at a prebuilt binary (the nix result, a container image), not only at a fresh go build. Separately, replace the fixed-behaviour fakemcp with a scriptable mock so each test states the server behaviour it needs.

Original notes

• testing
  • a general goal is to ensure all features have one or more integration tests
    • the mcp servers can be specialized mocks but the app entrypoint or binary is used and config is passed in or config path set to appropriate test
    • for both binary && app entrypoint (consider binary a superset)
      • all tests should pass for nix flake & container

Current state (main d0b7e01)

Binary.

• testsupport.MCPXBinary always runs go build -o <tmp>/mcpx ./cmd/mcpx once per test binary (internal/testsupport/build.go:54-73).
• No override exists to test a prebuilt binary. So in CI "the binary" means a fresh plain go build, never the artifact users install. The installed artifact is result/bin/mcpx: a makeBinaryWrapper C wrapper that puts nix-pinned git/deno/bun/node on PATH (package.nix postInstall) in front of .mcpx-wrapped. There is now a second shipped artifact, the binary inside Phase 5: tag event bodies must travel with the event #182's pkgs/mcpx/Containerfile image.
• So there are three drivers, not two (from Phase 2: user subject for tags needs an identity event #160): in-process ⊂ built binary ⊂ installed output. Only the installed output exercises the wrapper's PATH pinning, the ldflags version (internal/testsupport/build.go:63 builds without -ldflags, so mcpx --version differs between the test binary and the package -- Phase 3: range read_flight buffers frames with no cap while more is set #178), the generated man page and the completions. That rung should be a small suite -- a dozen assertions, not the whole e2e run -- and it is where Update Rust crate rand to v0.9.3 [SECURITY] #134's nix-build leg and Phase 3: range read_flight buffers frames with no cap while more is set #178's assertions run.
• The temp dirs from os.MkdirTemp (build.go:28,56) are never removed.

Entrypoint. main() (cmd/mcpx/main.go:52-204) does all of the following, and none of it is reachable from a test:

• reads os.Args;
• handles global flags inline (--json, --profile, --config, --tui, ...);
• builds &cli.App{Version, Paths: daemon.ResolvePaths()};
• sets the package global cli.SetPlumbingSource(app) (internal/cli/scripts.go:95);
• dispatches through a local handlers map (main.go:119-162);
• exits through os.Exit at five sites (main.go:74,103,179,188,202).

Process-global state that an in-process runner would share between tests:

• daemon.ResolvePaths reads os.Getenv (internal/daemon/paths.go:34-52);
• settings.Environ() is os.Environ() (internal/settings/bind.go:218);
• internal/cli non-test files reference os.Stdout/Stderr/Stdin/Exit/Getenv 86 times.

Autostart re-executes itself. EnsureDaemon spawns autostart.bin or, if that is unset, os.Executable() (internal/cli/client.go:281-287). In-process, os.Executable() is the test binary, which would be started as a "daemon". An in-process runner must set autostart.bin to the built mcpx, or use daemon.inline=true (internal/settings/registry.go:379).

Mocks.

• fakemcp (internal/testsupport/fakemcp/main.go, 321 lines) answers from compiled-in behaviour, with two knobs: FAKEMCP_FAIL_START and FAKEMCP_START_DELAY (main.go:35,39).
• askmcp (272 lines) exists separately for elicitation (build_proto.go:19-39).
• A new behaviour means editing a shared binary that every test uses.

Config. Each test writes .mcpx.json into its temp dir, with FAKE replaced by the fakemcp path, and sets MCPX_CONFIG (internal/e2e/e2e_test.go:41-60). That already matches "config path set to appropriate test". "Config passed in" (--config, main.go:88-94) is a separate path that nothing tests specifically.

Coverage. e2e contributes nothing to coverage, because the binary is built without -cover. Since Go 1.20, go build -cover produces a binary that writes coverage data to $GOCOVERDIR on exit. go tool covdata percent|textfmt -i=<dir> reads it, and go test -cover ./... -args -test.gocoverdir=<dir> writes unit coverage into the same directory so the two merge. Source: https://go.dev/doc/build-cover. A process killed by a signal writes nothing, so a detached daemon killed by stop must exit through its normal path for its coverage to count.

Approaches -- entrypoint

A -- in-process only (least recommended)

Replace the subprocess with cli.Run everywhere. This is the fastest option, and it loses exactly what an integration test is for:

• real exit codes and signal handling (signal.NotifyContext, main.go:53);
• argv parsing as the shell delivers it;
• the detached daemon (Setsid, client.go:312);
• the process-group kill of runtimes (runner.go:413-432);
• the nix wrapper's PATH;
• environment inheritance: Update dependency htmx.org to v1.9.12 #138 is a bug that exists only in the subprocess path.

Shared process globals would also make in-process tests interfere with each other. Not recommended.

B -- binary only, instrumented

Keep today's suite and add two things: a -cover build of the binary under $GOCOVERDIR, and an override (MCPX_TEST_BINARY=/path/to/mcpx, name to be decided) so the same suite runs against result/bin/mcpx or inside a container. It needs almost no code change, gives coverage, and makes "tests pass for nix flake & container" mean the shipped artifact (#134). It does not help speed (each call still pays a fork+exec plus daemon start) or debugging (a breakpoint in a subprocess needs dlv attach).

C -- both, with the binary suite as a superset (recommended)

1. Refactor: func Run(ctx context.Context, args []string, env Env, stdio Stdio) int in internal/cli, where Env is an environ slice or lookup function and Stdio is {In io.Reader; Out, Err io.Writer}. main() shrinks to signal setup plus os.Exit(cli.Run(ctx, os.Args[1:], os.Environ(), cli.Stdio{...})). Global-flag parsing, looksRunnable and the handlers table move with it. The table becomes an exported cli.Commands(), which Update dependency @chenglou/pretext to ^0.0.9 [SECURITY] #132's guard needs.
2. Thread the environment in stages. Stage 1: Run takes env but internals still read os.Getenv, so in-process tests use t.Setenv and cannot use t.Parallel. Stage 2: ResolvePaths, settings.Environ and plumbingApp take their inputs from Run, so in-process tests can run in parallel. That is the real speed gain, and it pairs with the CI time budget in Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95.
3. Harness: newEnv gets a runner interface with binary and inproc implementations. A test says what it needs, for example e := newEnv(t, cfg, needsProcess). The binary suite runs every test (the superset). The in-process suite runs the ones that do not need a process. The daemon stays a real process in both, via autostart.bin set to the built binary. An in-process daemon in a goroutine would leak goroutines and listeners between tests, which is the in-process version of chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52 (fixed for the subprocess path in Phase 5: tag event bodies must travel with the event #182).
4. Binary override and -cover from B, so the binary suite can target the nix result or a container image and produce coverage.

Recommended because it matches the notes (binary a superset) and makes the fast path the default for day-to-day agent runs (#99). It also avoids building a second harness. Its risk is the size of the refactor. Stage 1 is small; stage 2 touches every os.Getenv in internal/cli and internal/daemon.

The user's framing is right. One refinement: "app entrypoint" should mean cli.Run, not cli.App methods called directly. Calling app.CmdExec directly skips global-flag parsing and SettingsErr (main.go:183-189), so it would test a code path users never take.

Approaches -- mock MCP servers

A -- ad-hoc mocks per test (least recommended)

Each test that needs an unusual server writes its own Go program or httptest server. It is flexible, but it multiplies binaries to build (each is a go build, about 1 s) and duplicates the protocol plumbing each time.

B -- more knobs on fakemcp

Keep adding FAKEMCP_* environment variables. This is the path today. It stays cheap until the knobs interact, and then the fake becomes a second product with its own bugs.

C -- one scriptable mock (recommended)

fakemcp reads an optional behaviour file (--script <path> or FAKEMCP_SCRIPT), which declares:

• tools and their schemas and results;
• delays and failures, per call number;
• server-initiated requests (elicitation, sampling, list_changed notifications);
• the protocol revision to claim.

With no file it behaves exactly as it does today, so the existing e2e tests (199 at d0b7e01, 209 at 05c78b2) are unchanged. askmcp folds into it later, or stays if its opposite-behaviour rationale (build_proto.go:19-22) still holds. The script sits next to the test that uses it. The same file format can drive a remote mock (#134): the fake is baked into a container and gets its script over a mounted path. A behaviour that randomises (for example "fail 1 call in N") takes the run seed (#93).

Config: keep MCPX_CONFIG as the default and add one test each for --config <path> and for .mcpx.json found by the directory walk (internal/config/config.go:549-565), so all three ways config reaches the binary are covered once. When MCPX_CONFIG is unset, the walk also reads user-level files, so the walk test needs HOME/XDG_CONFIG_HOME isolated (#138).

Open questions

• Which runner is the default for a new test: in-process, with the binary run in CI only? Policy for agents belongs to Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99.
• What should the override be named (MCPX_TEST_BINARY, or a -mcpx.binary test flag)? A test flag does not leak into child processes, which is an argument for it.
• --tui and bare mcpx need a TTY. They are binary-only, or run under a pty via github.com/creack/pty (a new test-only dependency).
• Should the in-process suite count toward coverage at all, given that -cover on the binary already covers the same code?
• Does a -cover binary run slowly enough to trip the daemon-start timeouts that internal/testsupport/build.go warns about? Measure before making it the CI default (from Phase 2: user subject for tags needs an identity event #160).
• Where do the GOCOVERDIR artifacts go in CI, and is there a coverage floor? That belongs to Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95 and Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99.

Ease · Impact · Related

Ease L (stage 1 plus override plus -cover: M; stage 2 env threading: L; scriptable fake: M) · Impact high
Depends on / related: #132 (exported command table), #134 (prebuilt-binary runs per environment), #138 (env isolation is a subprocess-path property), #79 (faster CLI/exec; the same main.go code), #178 (the ldflags version the installed-output rung checks), #52 (closed, fixed in #182), #93 (seeded mock behaviour), #95 (coverage artifacts, parallel shards), #99 (which runner agents use).

Security/auth deferred; see tracker.

Acceptance criteria

• An in-process entrypoint exists beside the binary suite: the dispatch in cmd/mcpx/main.go is reachable as cli.Run(ctx, args, env, stdio) int (or the equivalent shape the implementer pins on this issue), and most integration tests can run in-process against it.
• The binary suite is retained as the superset (signals, exit codes, argv parsing, detached daemon, nix wrapper) and can target a prebuilt binary (nix result, container image), not only a fresh go build.
• The fixed-behaviour fakemcp is replaced or supplemented by a scriptable mock MCP server whose behaviour each test states.
• The work described in the Summary/Scope above is delivered via a branch + PR (no direct push to main); verification runs in GitHub Actions CI, not on the sandbox.
• Evidence (CI links, timings, or probe output as applicable) is recorded on this issue before it is closed.