mcp: global stdio servers spawn once per loaded directory and exhaust memory #51003

Activity

1. Nasnl commented on Sep 25, 2026

   

   Nasnl

   on Sep 25, 2026

   More actions

   Additional Windows evidence for the per-location fleet: respawns are driven by a
   repeatable eviction cycle, and a malformed phantom location gets a full fleet on
   every cycle.

   Environment: Windows, OpenCode Desktop/CLI 2.0.16 (service opencode-cli.exe serve --service), 5 configured local stdio MCP servers, plugins enabled.
   Evidence from ~/.local/share/opencode/log/opencode.log
   (grep location services evicted / mcp connected).

   1. Per-directory eviction runs on a ~61-minute cadence (UTC, same directory):

   evicted at	delta
   07:30:04.673	—
   08:31:05.593	+61m00.9s
   09:32:20.354	+61m14.8s
   10:33:35.022	+61m14.7s

   The user-home directory shows the same +61m07s / +61m15s pattern. Nothing
   external precedes any eviction in the log — it reads like an internal TTL sweep.

   2. Each eviction rebuilds that directory's full MCP fleet within ~16s:

   09:21:12.071  location services evicted directory="C:\Users\<user>"
   09:21:12.133  location services evicted directory="\\"
   09:21:12.401  watcher stopped ...
   09:21:12.813  configuration normalization ...
   09:21:12.936  location services booted directory="\\"
   09:21:21.037  mcp connected server=<one>
   ...            (all 5 local MCPs reconnect by 09:21:28)
   

   So the fleet is fully replaced per directory every cycle — matching the
   "groups of five" console flashes reported upthread on PR #45259 (each burst
   reconnects exactly 5 local MCPs through the cmd.exe/npx chain).

   3. Phantom location directory="\\" (bare backslash) gets the same treatment:

   • first booted 2026-09-22 06:14:34Z with workspaceID=undefined; 14 boots in
     the log; participates in every eviction/reboot cycle like a normal location
   • every boot is preceded within milliseconds by watcher subscribe path="C:\" type=entries → location services booted directory="\\"
   • receives all 5 local MCPs on every boot

   So per-sweep process count = directories actually in use + this phantom. The
   watcher correctly targets C:\ while the location key comes out as \, which
   looks like a root-path normalization issue. Worktree.DirectoryUnavailableError
   failures run through the same bootstrap but timing is inconsistent, so I can't
   establish causation from logs alone.

   Ask: any way to stop the eviction-cycle respawn for idle locations, and any way
   to suppress/deregister the "\\" location. I can provide redacted log excerpts
   or run a verbose-trace startup to capture who first requests it.
2. Kirytho commented on Sep 27, 2026

   

   Kirytho

   on Sep 27, 2026

   More actions

   Data point from 2.0.18 (released), which reframes this as a broken invariant rather than only a scaling concern.

   Environment: 2.0.18, Linux, opencode serve --service, 6 local stdio MCP servers + 1 remote, one TUI, one directory.

   After a single client disconnect/reconnect cycle I ended up with two complete sets of the 6 local servers — 12 processes, ~789 MB RSS — where the service reports only 7 live connections (opencode mcp list). Both sets are annotated directory=/home/fer and workspaceID=undefined, i.e. the same canonical Location key per canonical() in location-service-map.ts:39-44. Two mcp connected bursts, 2 minutes apart, the second immediately after the client reconnected. Zero session expired / reconnecting events, and only 2 teardown events in the entire log, so the first set was never disposed.

   The part I think matters most: packages/core/src/mcp/stdio.ts:39-40 documents an explicit guarantee —

   The process is acquired in the calling scope: closing the scope kills it (the spawner kills the whole process group, so descendants go too) regardless of whether the transport was closed.

   If a second set exists for a Location whose scope was already released, that guarantee is not holding. That would make this a correctness bug in the scope/finalizer path, not only the per-Location cost model that this issue and #37844 describe.

   I checked the paths that could plausibly own this and they all look correct: replaceServer (mcp/index.ts:493) disposes the previous entry, connect (:633), reconnect (:578) and recover (:333) each call stopServer before startServer, and startServer closes its forked scope on the failure branch (:458). So I could not find the culprit by reading, only the symptom.

   What I could not determine: I was unable to authenticate to GET /api/debug/location, so I cannot confirm from the server's own view whether this is one Location built twice or two Locations. If someone can run that endpoint after a reconnect cycle, connectionID in the log lines distinguishes the builds.

   Not filing a separate issue — this looks like the same bug.
3. Kirytho commented on Sep 27, 2026

   

   Kirytho

   on Sep 27, 2026

   More actions

   Correction to my comment above — I found the trigger, and it is not what I described.

   I claimed the duplicate set came from a client disconnect/reconnect. That was wrong. The trigger is opencode reload: every reload re-boots the Location layer graph, and the previous boot's MCP subprocesses are not reaped.

   Evidence from a cold opencode service restart with no client churn at all:

   14:45:37.032  location services booted  directory=/home/fer workspaceID=undefined durationMs=88
   14:45:42.228  cli starting args="[\"reload\"]"          <- reload
   14:45:42.262  location services booted  directory=/home/fer workspaceID=undefined durationMs=23
   14:45:42.262  location services reloaded count=1
   

   A single reload took the process tree from 6 local MCP servers to 12 (~890 MB). Across the whole log: 25 reloads, 51 location services booted, and only 2 event stream disconnected. The boots track reloads, not reconnects.

   Two consequences:

   1. This is already reported, and more precisely, in server: stdio MCP child processes not terminated on service restart / config reload #50758 ("stdio MCP child processes not terminated on service restart / config reload"). That title names the trigger outright. I had it in my search results while investigating and did not connect it. Pointing there rather than here.

   2. It contradicts the framing in serve (1.18.25): MCP child processes accumulate on web-client reconnects until the server OOMs #46035 ("accumulate on web-client reconnects") — at least for the serve path. Reconnects were only 2 in my log; reloads were 25. If reconnects are the trigger there, it is a different code path from this one, and worth separating.

   Also worth noting for whoever picks this up: the reload rebuilds plugins as well as MCP — plugin reconciliation started appears 4 times for 2 boots — so this is the Location layer graph being rebuilt, not an MCP-specific leak. The stdio.ts:39-40 guarantee ("closing the scope kills it... regardless of whether the transport was closed") is still the right thing to check, but the scope being closed is the reload path, not a client disconnect.

   Retracting the reconnect hypothesis from my previous comment.
4. kernel-oops commented on Sep 27, 2026

   

   kernel-oops

   on Sep 27, 2026

   Contributor

   More actions

   Another data point, plus a workaround that has held up for us (v2.0.16 fork, single long-lived serve on Linux, busy multi-agent use with many git worktrees).

   Scale. After about 45 hours the server had 50 loaded locations (one per worktree or project directory any session had touched) and therefore ~150 stdio MCP helper processes for three global local servers (50 + 50 + 49). Server RSS was ~10 GB, CPU sat at 100–120%, and the API periodically stopped responding altogether for up to ~2 minutes (one captured freeze: no request served for 2m20s), while agents carried on afterwards.

   Workaround. Run each stdio helper once, behind a small loopback bridge that speaks streamable HTTP, and point the global config at it as a remote server:

   "image-viewer": { "type": "remote", "url": "http://127.0.0.1:4411/mcp", "oauth": false, "enabled": true }

   The bridge initialises the helper once, answers each client's initialize from that result (echoing the client's protocol version), and remaps JSON-RPC ids so concurrent locations never collide. For plain tool servers (no server-initiated notifications or sampling) that is all that's needed. After switching, every location connects to the same three processes instead of spawning its own; RSS after restart was ~0.5 GB.

   Caveat, consistent with the reload finding above: make the config change while the server is stopped. Changing MCP config on a live server reloads the locations and the previous helper processes are not reaped.

   A proper fix would still be preferable: sharing one instance of each global stdio server across locations (reference-counted, torn down on config reload), or spawning lazily and releasing with the location's idle expiry.
5. titoune33 commented on Oct 14, 2026

   

   titoune33

   on Oct 14, 2026

   More actions

   Data point that narrows the leak (macOS, current dev).

   @Kirytho's "every reload re-boots the Location layer graph and the previous boot's MCP subprocesses are not reaped" is the right trigger. But the reaping contract at the RcMap/LayerMap layer is actually sound — it does not orphan the entry scope on reload. I added a runnable probe (PR #55548) and it passes both paths on dev:

   • idle entry (refCount 0, the "cold restart, no client churn" repro): reload() → invalidate closes the old entry scope eagerly.
   • borrowed entry (refCount > 0): invalidate detaches it, and the entry scope closes when the last borrower releases (RcMap.release sees the detached entry and calls closeEntry).

   Relevant ownership: Mcp.node is a location node (packages/core/src/mcp/index.ts → makeLocationNode), included in Instance.layer, and each stdio server forks its scope from the location entry's root (Scope.fork(root), killed on scope close). So the fleet should die with the entry scope — which, per the probe, does close on reload.

   So the leak is not "reload leaves the entry scope alive." It's one of:

   1. A stale borrow — some caller holds locations.get(ref) / contextEffect(ref) in a scope that outlives the reload, so the old entry's refCount never returns to 0 and closeEntry never runs. Each reload then strands one fleet behind that permanent borrow.
   2. The MCP reconnect path — config.updated → mcp.reload() re-forking from root without closing the prior per-server scope.

   Suggest the next repro step is to instrument the refCount of the detached entry across a reload (log before/after invalidate), or to GET /api/debug/location and check whether the old location is still listed as loaded after reload. If it is, it's (1) and the fix is the borrow discipline, not the layer.
6. mlimarenko commented on Oct 15, 2026

   

   mlimarenko

   on Oct 15, 2026

   Contributor

   More actions

   @titoune33 The probe result you report helps separate the reload investigation from #53046. That PR defers MCP startup and releases discovery-only connections after 60 seconds; a connection used for a tool, prompt or resource read, or explicitly connected, stays retained until disposal. It does not change Location borrow ownership, so it should not be treated as a fix for a stale borrow across reload.

   In the rebased code, MCP replacement still follows disposeServer → stopServer → Scope.close before starting the replacement. I have not reproduced either proposed reload leak path, and #53046 does not claim to fix #50758. Recording the old entry's borrow count alongside per-server scope closure during a reload would distinguish those paths without assuming that invalidation itself leaves the entry scope open.
7. rekram1-node

   self-assigned this

   on Oct 15, 2026
8. CerisonAutomation commented on Oct 22, 2026

   

   CerisonAutomation

   on Oct 22, 2026

   More actions

   Confirmed on opencode 2.1.2 (Homebrew), macOS 27.0.1.

   With exactly two MCP servers configured (caveman, supabase), the server spawns a separate set of child processes per loaded directory.

   mcp connected events, grouped by server and directory, from a single server log:

   12  server=supabase  | /Users/cb
   10  server=supabase  | /Users/cb/Downloads/dyad-1.16.0
    6  server=supabase  | /private/var/folders/.../T/opencode
    7  server=caveman   | /Users/cb
    5  server=caveman   | /Users/cb/Downloads/dyad-1.16.0
    5  server=caveman   | /private/var/folders/.../T/opencode
   

   45 connect events for 2 configured servers across 3 directories. In ps this showed up as 9 concurrent children under a single opencode serve --service PID — 3 x caveman + 3 x npm exec mcp-remote + 3 x node mcp-remote.

   Two amplifiers I found while measuring this:

   1. The npm exec wrapper. With command: ["npm","exec","mcp-remote","--","<url>"] each spawn costs an extra npm wrapper process (137-187 MB RSS per location in my measurement) on top of the node child. Pointing command at the resolved binary instead — node <path>/node_modules/mcp-remote/dist/proxy.js <url> — removes the wrapper entirely: 9 processes / ~800 MB -> 9 processes / ~447 MB, with identical behaviour (verified via mcp connected plus a successful tool round-trip).

   2. One of the per-location directories is RAM-backed. It is the agent scratch dir under /private/var/folders/.../T/opencode, so that instance's working set competes with the compressor for the same physical pages. That is why this presents as system-wide memory pressure rather than as one hot process. See [FEATURE]: Don't default agent scratch dir to a RAM-backed /tmp #55006.

   Practical note for anyone triaging this: killing duplicates blindly breaks the live session. The reliable attribution method is to snapshot per-PID CPU time, issue one tool call that does real work through the server, snapshot again, and diff — the instance whose CPU time moved is the one serving the session; the others are idle leftovers.

   A fix that reuses one stdio server process per (server config, workspace) pair instead of per location would remove the fan-out entirely.
9. btriapitsyn commented on Oct 22, 2026

   

   btriapitsyn

   on Oct 22, 2026

   More actions

   I maintain OpenChamber, the client in this report. I dug into a similar report on our side, openchamber/openchamber#5412, and here's what we found and changed.

   @ianrelecker the OpenChamber half of your setup is fixed. Since 2.5.0 we release a chat's location 30 s after you leave it and a worktree's after 5 min, through DELETE /api/debug/location. We also stopped reading a left directory's MCP status after its shutdown, which used to start it again within a second. The directories you saw re-touched after eviction were most likely us.

   What remained was the directory on screen. On location.shutdown OpenChamber bootstrapped it again about 30 ms later, so an app left open overnight brought it back with every MCP server once an hour, and each remote server went through OAuth with an expired token. In our next release it waits until the window is focused again. Projects get released too now, 15 min after the user leaves them. Our diagnostic export, Cmd/Ctrl+Shift+L, now shows how many locations OpenCode has loaded and how many processes it started, so the next report like this comes with those two numbers.

   I also tried to catch eviction leaking processes and couldn't. I put a fake stdio MCP behind a bash wrapper, the way uvx starts python, and evicted its location mid-handshake at delays from 0 to 2 s, after connect, and with an /api/shell command running: zero survivors on 2.0.32 and on 2.1.2. 60 evict/boot cycles with remote servers kept opencode serve between 275 and 330 MB.

   So the client can release locations sooner, and that's about all it can do. The memory goes into one copy of every global stdio server per location, and sharing identical ones across locations, as #37844 lays out, is the part only OpenCode can fix. The user in our report had 11 project and worktree directories open, which meant 11 copies each of codegraph and uvx mcp-proxy-for-aws.
10. mlimarenko commented on Oct 23, 2026

    

    mlimarenko

    on Oct 23, 2026

    Contributor

    More actions

    @btriapitsyn Your client-side release changes and #53046 address different parts of the fan-out. That PR defers configured MCP startup until MCP use and closes discovery-only connections after 60 seconds, keeping their catalogs cached. Tool, prompt or resource use, an elicitation, or an explicit connect retains the connection until disconnect or disposal. It does not share a process across locations, so directories that genuinely use the same server still get separate instances.

    @CerisonAutomation That distinction also applies to the per-directory copies you measured: #53046 can reduce discovery-only residency, but it is not global stdio pooling or a fix for every reload/process-cleanup report. I have not reproduced your macOS setup or the memory figures in these comments.