Design: mcpx client session & stream recovery — re-initialize on a legacy session 404, re-issue a broken modern request once #1634

Summary

mcpx's client has no recovery from the two ways a live upstream connection goes stale. When a legacy HTTP upstream restarts and forgets the session, every later call 404s forever — the client never re-initializes, and the pool cannot reap an HTTP client that never dies, so only a daemon restart heals it. When a modern (2026-07-28) upstream's response stream breaks, the in-flight request is reported lost and never re-issued, though the 2026 spec makes re-issue a client MUST. This design puts both recoveries in Client (the layer that owns the handshake, the pending table and the era): forget + re-initialize + retry once on a session-carrying legacy 404, and re-issue once with a new id on a broken modern stream — with tools/call re-issued only on annotation evidence, so a destructive tool is never silently executed twice.

Design doc: files/designs/code-mcpx-client-session-stream-recovery.md in the keep-the-four-repos-healthy goal workspace (full audit, alternatives, risks, open questions).

No Drew direction decision needed: the roadmap row is at proposed, both behaviours are spec MUSTs (TR-29; 2026 changelog / PM-06), and the judgment calls (client-layer recovery; the annotation gate) are recorded in the design doc.

Current state (verified on main ed8e591, 2026-10-28)

• The two gaps are open in the conformance register, pkgs/mcpx/internal/conformance/coverage_transport.go:114-115: streamable-http-client-reissue-broken-request and streamable-http-client-reinit-on-404, both client-side, origin mcpx client: legacy Streamable HTTP and stdio robustness — session 404, GET stream, SSE retry, string ids, stream timeout, version header dezren39/darwin-nix#203.
• 404 path: HTTPTransport.Send (internal/mcpclient/http.go ~L166-180) returns any non-2xx as HTTPStatusError (except a JSON-RPC error body naming the request, pushed as the reply). Nothing in Client treats a session-carrying 404 as expiry. Register TR-29 (docs/compare/register/transports.md:550-565): a restarted remote server "leaves its mcpx instance failing every call"; legacy spec (2025-03-26..2025-11-25): a client receiving 404 for a session-carrying request MUST start a new session with a fresh initialize.
• Broken-stream path: a modern SSE response stream that ends unanswered calls t.lost(reqID) (http.go ~L190-205), synthesising a -32000 error (lost/codeStreamLost ~L378-389); the caller just fails. Legacy streams instead resume via GET + Last-Event-ID (resume(), http.go ~L338) — legacy has resumability, 2026 deliberately does not (PM-06, docs/compare/register/process-model.md:133-146).
• Reusable machinery: Client.roundTrip (internal/mcpclient/client.go ~L953) is the single request choke point; initializeLegacy/finishLegacy (~L343-388) already validate the version, refresh ServerInfo/Capabilities and restart the GET listen stream; the catalogue-invalidation signal (invalidate → Notifications.OnListChanged, ~L683-749) already reaches the pool; Tool.Annotations (~L159-170) carries readOnlyHint/idempotentHint/destructiveHint.
• Distinct from design(mcpx): resume upstream HTTP sessions after a cold restart — persist the resumable session state #1414: that design resumes a still-live session after our daemon restarts. This one handles the opposite event — the upstream restarted and the session is dead; the move is discard + new session, not resume.

Proposal (decisions recorded in the design doc)

• Recovery lives in the client. Not the transport (protocol-free; initialize is a client exchange) and not the pool (reaping loses the in-flight call, and nothing marks an HTTP client dead for the pool to reap).
• Legacy 404 (Phase 1): trigger only when a request (not a notification, not initialize) 404s, era is legacy, and the request carried a session id. The transport gains a report-and-clear session seam (the setNegotiated interface-assertion pattern); re-initialize is singleflight across concurrent calls (session generation — one initialize per upstream restart); the request retries once with a new id; a second 404 surfaces. After re-initialize, all three catalogue kinds are invalidated through the existing signal.
• Modern broken stream (Phase 2): the stream-lost error gets a stable identity (not message matching); the request layer re-issues once with a new id. tools/call is gated: automatic re-issue only when the tool's list-time annotations declare readOnlyHint or idempotentHint (PM-06's double-execution hazard is the register's own warning); otherwise the stream-lost error surfaces and the host decides. Legacy is excluded — resumability is its mechanism.
• Truth-up (Phase 3): both conformance gaps retired with test evidence; TR-29 and PM-06 register rows rewritten as-built; the roadmap's "Legacy client robustness" row updated to name the umbrella's remainder; evidence comment on mcpx client: legacy Streamable HTTP and stdio robustness — session 404, GET stream, SSE retry, string ids, stream timeout, version header dezren39/darwin-nix#203 (comment, not close — the umbrella is its owners').

Phases (each independently landable; filed as sub-issues)

1. Legacy session-404 re-initialize + retry once — transport seam, singleflight client recovery, one retry, catalogue invalidation; gap streamable-http-client-reinit-on-404 retired.
2. Modern broken-stream re-issue, annotation-gated — stable stream-lost identity, once-only re-issue with a new id, annotations retained at list time, tools/call gate; gap streamable-http-client-reissue-broken-request retired. Independent of Phase 1.
3. Register truth-up + nix#203 evidence — TR-29 / PM-06 rows as-built, roadmap row updated, evidence comment on mcpx client: legacy Streamable HTTP and stdio robustness — session 404, GET stream, SSE retry, string ids, stream timeout, version header dezren39/darwin-nix#203. Gated on Phases 1–2.

Acceptance (design level)

• A legacy upstream that restarts mid-life costs at most the calls in flight: they recover through exactly one re-initialize and one retry; later calls unaffected; no daemon restart involved.
• A broken modern stream loses no read-shaped request (re-issued once, new id); a tool call is re-executed only on annotation evidence, else its caller gets the stream-lost error.
• Both conformance gaps retired with tests; TR-29 and PM-06 tell the truth; mcpx client: legacy Streamable HTTP and stdio robustness — session 404, GET stream, SSE retry, string ids, stream timeout, version header dezren39/darwin-nix#203 carries the completion evidence.

Acceptance criteria

• Legacy recovery is specified and landed via the phase issues (mcpx: Phase 1 of #1634 — re-initialize on a legacy session 404 and retry the request once #1637, mcpx: Phase 2 of #1634 — re-issue a broken modern request once (tools/call gated on annotations) #1638, mcpx: Phase 3 of #1634 — register truth-up (TR-29, PM-06) and nix#203 evidence #1639): on a session-carrying legacy 404 the client forgets the session, re-initializes, and retries the request exactly once.
• Modern recovery: on a broken 2026-07-28 response stream the in-flight request is re-issued exactly once with a new id; tools/call is re-issued only on annotation evidence, so a destructive tool is never silently executed twice.
• The two conformance-register gaps named in the design (legacy session 404, broken modern stream) are closed in the register when the phases land.
• No daemon restart is required to heal either stale-connection case.