mcpx docs: markdown emitters in the binary and a drift check for every generated page #1846

Relocated from dezren39/darwin-nix#145 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #96

Summary

Implement the generated reference as #144 recommends (Approach C). Existing commands gain markdown formats. The renderers sit next to the renderers that already exist (Describe, JSON, OpenAPI, ManPage). Tables that live in committed docs are regenerated in place between markers and guarded by a golden test. Site-only pages are generated at build and never committed.

This issue was added by the architect. The user's notes have no "generated reference" bullet. It exists so the pages the notes ask for ("env vars", "cli arguments", "cli commands", "mcp compatibility", "/v1") and the compendium's reference sections cannot drift from the binary. #144 records that docs/parity.md has already drifted: 27 of 60 ops have no row.

If #144 picks a different approach, rewrite this issue before starting.

Original notes

• documentation
  • make a book using whatever is the most common and simple modern choice
    • 3 main things, not book sections jsut like levels of detail-
      • the 'ok i just installed this thing what even is it' layer
        • a page or section in a page for
          • env vars
          • cli
          • cli arguments
          • cli commands
          • mcp
          • mcp compatibility
          • /v1
      • the compendium
        • basically everything that is built can be described here in as much detail as you want. you don't need to hold back it's an encyclopedia

Current state

See #144 "Current state" for the full inventory with file references. In short, from pkgs/mcpx/ on main (line numbers re-cited at 05c78b2 in the 2026-09-30 issues pass):

• Settings. The "Every setting" table in docs/configuration.md:149-474 was produced by a generator that is not in the tree. internal/settings/docs_test.go:19-48 checks only that each path and MCPX_* name appears.
• Commands.
  • internal/cli/manual.go:33 Commands() has 32 entries. Dispatch (cmd/mcpx/main.go:119-162) also has serve, adapter, registry, openapi, api, man and completion.
  • Top-level help is a hand-written const (internal/cli/root.go:549-611).
  • Global flags are a switch (main.go:62-104).
  • 130 per-command flags are hand-declared (118 x := fs.* plus 12 fs.Var), all through newFlagSet (root.go:34) and then parseFlags (root.go:52).
• /v1. api.Ops() (60 ops) → api.OpenAPI (internal/api/openapi.go:18). GET /v1/openapi.json adds live upstream tool paths (internal/daemon/ops.go:498-535).
• Parity. internal/api/parity_test.go checks code only and never reads docs/parity.md.
• Protocol. internal/mcpserver/revisions.go:292-316 (Features, FeatureMatrix), server.go:778 (Supported), server.go:1389 (capabilities(version, conn), a method on *Server; since Phase 2: delegated Manage cannot add members #171 it depends on whether the connection can push, and no longer declares logging).
• Man page. It stamps time.Now() (manual.go:335).
• Nix build. package.nix:88-94 writes the man page and completions. checkPhase (package.nix:73-81) does not include ./internal/api/ or ./internal/cli/.

Plan

The steps are ordered, and each can be its own commit. File paths are relative to pkgs/mcpx/.

0. Prerequisites: bug candidates from #144, land these first

These are not docs work, but publishing before they land publishes the bugs.

1. Complete the command table. Move the dispatch map from cmd/mcpx/main.go:119-162 into internal/cli, either as a Run field on Command or as a Handlers(*App) map. Add a test that every dispatched name is a Commands() name or alias. Declare logs as an alias of log.
2. Generate top-level help from Commands(). Replace the const usage (root.go:549-611), keeping its intro paragraph and the CONCURRENCY and EXAMPLES blocks as text. The CONCURRENCY block (:597-604) teaches the removed mode key and a nonexistent pooled value, so rewrite it to sharing/scope rather than keep it verbatim (Root flake check fails on nixosConfigurations due to missing clan-core.clanModules #159 item 2). Choose one set of groups (🎨 Palette: Improve accessibility of editor search panel #144 open question 6).
3. Make global flags data. Add GlobalFlags() []GlobalFlag in manual.go, and parse them in a cli.ParseGlobal(app, args) that main.go calls, so the parser, help and docs read one list.
4. Settle the acronym names before first publication: daemon.leaseTTL, proto.askTTL, proto.serveMCP and proto.stateTTL (configuration.md:216, :403, :407, :409). Either dashed() (internal/settings/schema.go:227-243) treats a capital run as one word, or those four get explicit Flag/Env values with the current spellings kept as FlagAliases/EnvAliases so nothing breaks.
5. Deterministic man date. Use SOURCE_DATE_EPOCH when set (manual.go:335). nixpkgs stdenv sets it (1980-01-01 unless the source provides one), so the Nix man page will show that date. A release-stamped date from amd host fails at evaluation: no outputHashes for p2panda-core-0.7.1 #158 / 🎨 Palette: Add ARIA labels and live regions to search panel #150 is the better long-term answer.

1. A markdown helper package

internal/refdoc/md.go is a leaf package with no mcpx imports. It provides:

• Table(headers []string, rows [][]string) string, escaping | and newlines in cells
• Code(s), with backtick-safe fencing
• Region(name, body), which wraps a body in <!-- ANCHOR: name --> … <!-- ANCHOR_END: name -->

mdBook strips any line containing ANCHOR: from an include, and GitHub hides HTML comments. So one file serves {{#include settings.md:daemon}} and still reads cleanly raw (#117, #120). The output must contain no generator syntax.

2. Settings

• Renderer. func (s *Schema) Markdown(opts MarkdownOptions) string in internal/settings/file.go, next to Describe (:175) and JSON (:219).
  • One ### <section> per top-level key, each wrapped in refdoc.Region(section, ...).
  • Keep today's columns (setting, kind, default, scope, hot, flag, variable, governs), so the first regeneration's diff shows only real drift. Render an empty default always as *(empty)*; configuration.md:166 currently renders it inconsistently.
  • Mark plumbing rows *(plumbing)* as today. List enum values.
  • opts.Long adds a definition list under each table with each setting's Long paragraph, for the site page. The committed table stays compact.
• Schema JSON. Add scope and hot to Schema.JSON (file.go:219-252). docs/configuration.md:152 already claims config --schema prints the same thing.
• Command. mcpx config --schema gains --format text|json|markdown (CmdConfig, root.go:296-379), with --json kept as --format json.

3. Commands and global flags

• Capturing hand-declared flags. parseFlags (root.go:52-67) gets a describe mode. While a package-level describing *flagCatalog is non-nil, it:

  1. binds the registry flags as today;
  2. records fs.VisitAll into the catalog under flagSetCommand[fs];
  3. returns errDescribed before any work.

  ManualMarkdown calls every handler with describe mode on and empty args. This works because every command already builds its flags through newFlagSet before side effects. Commands without flags (help, completion) record nothing.

  • Guard: a test that every handler returns errDescribed (or no flag set at all) within a short timeout and without touching the network or the daemon. A command that does work before parseFlags fails that test and must be reordered.
  • Subcommands: settings builds four flag sets all named "settings", and servers two named "servers". They need distinct names, such as "settings set", matching log sql and log record, which already do this. They also need describe-mode entry, either through a Sub []Command on Command (🎨 Palette: Improve accessibility of editor search panel #144 open question 4) or by calling each handler with each subcommand name as args[0].

• Flag rows. Each row has name and aliases, value type, default (fs.Lookup(name).DefValue) and usage. The source column says either "setting x.y (scope, hot)", linking to the settings page, or "this command". A hand-written flag that shadows a registry alias (mcpx daemon --log-level, root.go:105) is shown as the command's own, so the page states what actually happens.

• Renderer. internal/cli/manual_md.go: ManualMarkdown(version string) map[string]string, mapping a relative path to its content. It writes:

  Path	Content
  cli/index.md	The commands grouped with the man page's titles (manual.go:354-360, lifted to a shared var)
  cli/global-flags.md	The global flags
  cli/<name>.md	One page per command: usage, summary, aliases, detail, examples, flags
  settings.md	From Schema.Markdown with Long
  api.md, mcp-tools.md, protocol.md, parity.md	Steps 4-7
  SUMMARY.md	A fragment listing the pages above, which the docs build splices into the book's SUMMARY.md. mdBook renders only pages listed there.

• Command. mcpx man (CmdMan, root.go:255-276) gains --format roff|markdown (default roff) and -o DIR. With --format markdown -o DIR it writes the tree. With --format markdown and no -o it prints every page concatenated in SUMMARY order. That serves mcpx man --format markdown | less and the llms-full.txt input for Update module github.com/jackc/pgx/v5 to v5.9.2 [SECURITY] #120.

  No new command is added. man is "the manual", and the markdown form is the whole reference, of which roff is the terminal subset.

4. /v1

• Renderer. func Markdown(version string) string in internal/api/openapi.go, next to OpenAPI. It works from Ops() directly, not from the OpenAPI map, so that ToolName() / CoveredBy, Admin, Mutating, Destructive, Streams and Text survive.
• Layout.
  • Group ops by the first path segment after /v1/. Op has no group field; if the result reads badly, add a Tag.
  • Per op: a ### METHOD /path heading, summary, description, a parameters table (name, in, type, required, enum, description) and a flags line (admin, mutating, destructive, streams).
  • State that responses are untyped (all {"type":"object"}, openapi.go:110) rather than implying a schema.
• Raw specs. Publish both, statically. The static api.OpenAPI(version) goes on the site as reference/v1-openapi.json, emitted by ManualMarkdown or by mcpx openapi --surface daemon if 🎨 Palette: Improve accessibility of editor search panel #144 wants that flag. mcpx openapi output goes as reference/serve-openapi.json. Never fetch GET /v1/openapi.json from a running daemon: it carries the builder's configured tools (daemon/ops.go:498-535).

5. MCP tools mcpx serves

In manual_md.go: mcpserver.New(nil, "mcpx", version).WithExtras(mcpserver.OpTools(nil)).Tools(), the construction parity_test.go:90 already uses. Each tool gets name, description, annotations (readOnlyHint, destructiveHint) and a parameter table from the top level of InputSchema. Proxied tools link to their /v1 op.

6. Protocol tables

• Extract capabilities (mcpserver/server.go:1389-1420) into a pure DeclaredCapabilities(version string, push bool) map[string]any. The method then calls it with s.Notify != nil && c != nil && c.canPush(), the per-connection condition Phase 2: delegated Manage cannot add members #171 introduced.
• func ProtocolMarkdown() string in internal/mcpserver/revisions.go renders two regions:
  • features: the Features × Supported table from FeatureMatrix(). This is new; it is what the code consults.
  • capabilities: DeclaredCapabilities(v, true) per Supported version, replacing docs/protocol.md §2.2. A footnote states the push condition: the connection must be able to push, which HTTP connections cannot. The committed §2.2 still lists logging, which Phase 2: delegated Manage cannot add members #171 withdrew (Root flake check fails on nixosConfigurations due to missing clan-core.clanModules #159 item 18), so the first regeneration fixes a real error. docs/protocol.md is PROTOCOL's file; coordinate the marker migration with whoever owns it then.
• §1 (the revisions, eras and upstream defaults) and §2.1 (methods) stay hand-written, because the switch at server.go:504-765 is not data. Add a presence test that every Supported version has a §1 row and every case method string in server.go has a §2.1 row, using the same source-regex technique as parity_test.go:21.

7. Parity table

• Add two fields to api.Op, next to each op's declaration:

  • CLI []string: the commands that reach it, e.g. {"status"} or {"elicit answer"}.
  • Plugin string: the DaemonClient method name, e.g. alive.

  Tests check that every CLI entry resolves in Commands() (after step 0.1) and that every Plugin name matches ^\s*(async\s+)?<name>\( in plugin/opencode/mcpx/daemon.ts. This is the column Update Rust crate clap to 4.5.4 - autoclosed #76 wants to generate from as well.

• The renderer produces the /v1 table: Operation, Route, CLI, MCP tool, Plugin, and Notes derived from Streams/Admin/CoveredBy.

• The "CLI commands" table (parity.md:71-111) and "Known gaps" stay hand-written for now. The CLI table can be generated later by inverting Op.CLI, but its local-only rows (init, run of a named script) need a reason field.

• The 27 missing ops get rows on the first regeneration. Each needs CLI/Plugin filled in, or explicitly empty. SURFACES is changing docs/parity.md for Update Rust crate clap to 4.5.4 - autoclosed #76 now; whatever rows it adds by hand are replaced by the generated region when this lands.

• Owns Phase 3: sync, blob gate, and relays #181 collision 18: an absent row is a failure, not a gap. Today docs/parity.md cannot tell a known gap from a missing row, because its ## Known gaps prose covers only the 33 documented ops. The rule here:

  1. The generated region has exactly one row per Ops() entry, so a row cannot be absent without the golden comparison (step 8) failing. A separate presence assertion names the missing op in the failure, rather than printing only a byte diff.
  2. A known gap is always a row with an explicit value: an empty CLI renders as — plus a reason taken from a CLIGap string field (or equivalent) on the op, never as a missing row. An op with an empty CLI and no reason fails the test.
  3. The hand-written ## Known gaps prose may only name ops that have a row. A test checks each backticked op name in that section against Ops().

  The page's content (which reasons, which columns) belongs to whoever regenerates it (Update Rust crate clap to 4.5.4 - autoclosed #76, 🎨 Palette: Improve accessibility of editor search panel #144). This issue owns the check.

8. Committed docs: marker regions and the golden test

• Marker format. Each generated region in a committed file is bracketed like this:

  <!-- mcpx:generated settings (go test ./internal/refdoc -update) -->
  ...
  <!-- /mcpx:generated -->
  

  Everything outside the markers stays hand-written.

• Regions.

  File	Generated	Stays hand-written
  docs/configuration.md	The section tables, :160 to the end	The prose (:1-148) and the "Every setting" intro (:151-158). Fix the intro's claim to match what is now true.
  docs/parity.md	The /v1 table (:29-63)
  docs/protocol.md	The §2.2 table (:98-106), plus the new features table after §1

• Golden test. internal/refdoc/regions_test.go (package refdoc_test, which imports settings, api, mcpserver and cli, so there is no import cycle):

  • For each (file, region, renderer): read the file, extract the region, render, and compare.
  • On mismatch it prints a short diff and go test ./internal/refdoc -update.
  • With -update it rewrites only the bytes between the markers.
  • It fails on a missing or duplicated marker, so a region cannot be deleted silently.
  • Also add a //go:generate go test -run TestGeneratedRegions -update . directive in internal/refdoc/doc.go, so go generate ./... works too.

• internal/settings/docs_test.go.

  • Delete TestTheDocumentationListsEverySetting (:19-48). Byte equality of the region implies presence and also covers kind, default, scope, hot, flags, aliases and description.
  • Keep TestTheDocumentationDescribesTheScopes (:52-65). It checks prose outside the region, which the golden test does not cover.

• Determinism. A test renders ManualMarkdown twice and requires identical output, and requires the same output with MCPX_* variables set. The emitters read declarations, never resolved values. Map iteration anywhere in a renderer must be sorted.

9. Build wiring

• package.nix.
  • checkPhase (:73-81): add ./internal/refdoc/ and ./internal/api/, so nix build .#mcpx enforces the goldens and parity, not only CI's go test ./... (.github/workflows/mcpx.yml:65). Confirm that neither spawns runtimes or binds sockets: the sandbox cannot, as the comment at package.nix:69-71 says.
  • postInstall (:84-94): after the man page, add $out/bin/mcpx man --format markdown -o $out/share/doc/mcpx/reference, and copy the two OpenAPI JSON files there. Nix users then get offline docs, and the docs derivation can use them without rebuilding mcpx.
  • Man date as in step 0.5.
• Docs derivation (fix(deps): update github.com/charmbracelet/ssh digest to ebfa259 - autoclosed #118). .#mcpx-docs copies ${mcpx}/share/doc/mcpx/reference into the book's src/reference/, splices reference/SUMMARY.md into the book summary, and runs mdbook build. The generated pages are never committed.
• Prose includes. Hand-written pages (🎨 Palette: Add aria-labels to search and goto-line inputs #146 configuration, ⚡ Bolt: Optimize hash calculation for cursor colors #147 CLI and /v1) and compendium entries (docs(id): design plan, research, and n0 relay default #152) use {{#include ../reference/settings.md:daemon}} and the like, pointing at the generated directory, not at the committed marker copies.
• Agent twin (Update module github.com/jackc/pgx/v5 to v5.9.2 [SECURITY] #120). The generated pages are already plain markdown, and mcpx man --format markdown (concatenated) is a ready llms-full.txt section for the reference.

Docs checks run in CI; see the testing trackers under #91 (CI pipeline #95, integration/environments #97).

Verification

• go test ./internal/refdoc ./internal/settings ./internal/api ./internal/mcpserver, unit only. The e2e suite is not needed for this. It no longer leaks daemons (chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52, fixed in Phase 5: tag event bodies must travel with the event #182), but it is slow and tests nothing here.
• Break it on purpose: edit one Short in internal/settings/registry.go → the golden test fails and names -update; -update → only that row changes. Delete one /v1 row from docs/parity.md by hand → the test fails and names the missing op (collision 18).
• nix build .#mcpx, then check that result/share/doc/mcpx/reference/cli/serve.md exists (step 0.1) and that two builds produce identical result/share/man/man1/mcpx.1.
• mcpx man --format markdown -o "$d" on any machine produces the same bytes as the Nix output of the same commit.
• In the built site, every page in reference/SUMMARY.md renders, and every {{#include ...:anchor}} resolves (mdBook fails on a missing anchor).

Acceptance

• Every setting, command, command flag, global flag, /v1 op and served MCP tool appears in a generated page, from code, with no hand edit.
• docs/configuration.md, docs/parity.md and docs/protocol.md have marker regions that CI fails on when stale. parity.md lists all ops, and an absent row fails the check even when the rest of the table is correct (Phase 3: sync, blob gate, and relays #181 collision 18).
• mcpx config --schema --format markdown and the settings page are the same bytes.
• No page lists a flag that its command does not read. This is Update Rust crate petgraph to 0.6.4 - autoclosed #74's walk: Update Rust crate proc-macro2 to 1.0.81 - autoclosed #75 built it for exec's flags, and Add omasnap upstream repository to flake inputs #129 owns extending it to every setting (Phase 3: sync, blob gate, and relays #181 collision 15). Until it covers a command, per-command pages carry a scope column and a one-line explanation of daemon-scoped flags on client commands.

Open questions

1. Describe mode or declared flags, and how subcommands are entered (🎨 Palette: Improve accessibility of editor search panel #144 open questions 3 and 4)?
2. Should Op gain a Tag for grouping, or is the path prefix enough?
3. mcpx openapi --surface daemon: worth adding, or is writing v1-openapi.json from man --format markdown -o enough?
4. Should the man page date come from a release stamp (amd host fails at evaluation: no outputHashes for p2panda-core-0.7.1 #158) rather than SOURCE_DATE_EPOCH?
5. Is internal/refdoc the right name? internal/openapi already exists, for mcpx api --spec, so avoid anything OpenAPI-sounding.

Ease · Impact · Related

Ease L. The renderers are each small. The size is in step 0 (command table, global flags), describe mode across about 35 handlers and their subcommands, the parity fields for 60 ops, and marker migration of three docs. · Impact high. The reference cannot drift, Nix users get offline markdown docs, and the agent twin gets its reference section for free. · Depends on / related: #144 (design), #117 (mdBook includes and anchors), #118 (docs derivation), #120 (agent twin), #146, #147, #152, #158, #49 (CI for every package), #74 (closed, fixed in #75; flags accepted and ignored), #76 (CLI for /v1-only ops; shares Op.CLI; owns the parity page's CLI content), #77, #159 (the step-0 bugs), #52 (closed, fixed in #182).

Security/auth deferred; see tracker.