core: long-lived V2 server enters persistent allocation loop #36677

Activity

1. opencode-agent

   assigned

   simonklee

   on Jul 13, 2026
2. opencode-agent

   added

   bugSomething isn't workingSomething isn't working

   perfIndicates a performance issue or need for optimizationIndicates a performance issue or need for optimization

   coreAnything pertaining to core functionality of the application (opencode server stuff)Anything pertaining to core functionality of the application (opencode server stuff)

   2.0

   on Jul 13, 2026
3. simonklee

   removed their assignment

   on Jul 13, 2026
4. dondetir commented on Jul 22, 2026

   

   dondetir

   on Jul 22, 2026

   Contributor

   More actions

   Spent some time on this from the code side. Findings, in case they help narrow it down:

   Ruled out with evidence:

   • Boot failure retry loop: LayerMap/RcMap in effect 4.0.0-beta.83 cache a failed location build for the whole 60 minute idle TTL, they do not retry per request. Verified empirically against the same effect version.
   • Teardown fiber leak: the agent.updated / command.updated / catalog.updated emissions you observed at expiry are the State.create scope finalizers themselves (state.ts finalize), so location scope close provably runs. Boot emits the same three events when the state nodes first materialize.
   • The .updated to refetch to reacquire cycle is client driven and quiesces once the RcMap entry is warm; a warm get is a cache hit.
   • All literal spin constructs in core/server (while(true), Schedule.forever) block on queues or terminate. The session run coordinator goes fully quiescent when no wake is pending. Server SSE feed subscribers use dropping queues (4096) and self evict on overflow.

   Still standing, and it matches your "state accumulated across expiry/reboot cycles" observation: subscriber accumulation on the two process global unbounded PubSubs in packages/core/src/event.ts (pubsub.live and the per type map, which is never pruned). Location scoped services subscribe to these global pubsubs from at least 8 call sites. One concrete path with no guaranteed unsubscribe is the plugin bridge in packages/core/src/plugin/promise.ts, which hands plugins Stream.toAsyncIterable(events.subscribe()); if the async iterator is never returned/closed, that generation's subscriber survives on the global pubsub. At N leaked subscribers every published event fans out to N unbounded queues, which would show up exactly as steady JS allocation churn.

   If you can still reach a degraded server, dumping the subscriber counts of pubsub.live and pubsub.typed and comparing them against the number of live locations would confirm or kill that theory quickly.

   While digging I also found a separate correctness bug in the same lifecycle: a location whose boot fails once serves the cached failure for the full 60 minute TTL with no recovery path (production never calls locations.invalidate). Filed as #38405 with a repro, fix in #38406.
5. dondetir commented on Jul 23, 2026

   

   dondetir

   on Jul 23, 2026

   Contributor

   More actions

   Following up on my earlier analysis: I can now reproduce the accumulation mechanism deterministically in about 4 seconds.

   Setup: a promise plugin that consumes ctx.event.subscribe() in a detached for await loop and never closes the iterator, which is a normal thing for a real plugin to do. Cycle a location through boot then invalidate (invalidate exercises the same entry scope close as the 60 minute idle expiry) and count subscribers on the process global pubsub.live in packages/core/src/event.ts (the internal subscribers set size).

   Results over 10 cycles:

   • Control, no plugins: subscriber count returns to 0 after every cycle. The base location lifecycle cleans up correctly.
   • With the plugin: exactly one subscriber leaks per cycle, monotonically: 1, 2, 3, ... 10 after invalidate.

   Leak point: packages/core/src/plugin/promise.ts line 93, subscribe: () => Stream.toAsyncIterable(host.event.subscribe().pipe(Stream.map(wireEvent))). The subscription's lifetime is owned by the async iterator, not the plugin generation, so when a location generation is torn down the plugin's subscriber survives on the global unbounded PubSub. Nothing closes it: the detached loop keeps pulling, so the subscription stays registered on the global PubSub for the life of the process even though its location generation is gone.

   This lines up with the report, with one caveat on what's actually reproduced: the harness measures subscriber accumulation, not CPU or RSS. The accumulation is deterministic. The rest follows from it: each leaked subscription stays live and its detached loop keeps draining, so every published event is delivered to and processed by all accumulated generations. The cost is per-event fan-out across N live subscriptions plus the retained fibers and iterators, not an undrained queue backlog. A fresh process has none of this, which is consistent with a clean run staying under 1 percent.

   Fix direction I would propose: tie the promise bridge event subscription to the plugin generation scope so teardown interrupts the stream even when the plugin never calls return() on the iterator. Happy to open a PR for this referencing this issue.

   Repro harness (test file plus a small fixture plugin) available on request; it is 60 lines against packages/core/test.
6. tiffanychum commented on Jul 25, 2026

   

   tiffanychum

   on Jul 25, 2026

   More actions

   Re-triage note from PR #38825 (now closed):

   dondetir’s deterministic repro depended on a Promise plugin detaching for await on ctx.event.subscribe(), which leaked one process-global pubsub subscriber per location boot→invalidate cycle.

   On current dev, that API is gone. The namespaced plugin rewrite (909a1a6d7) replaced PluginContext without an event surface; Promise and Effect hosts no longer expose event subscribe. Remaining in-process EventV2 subscribers in built-in plugins use Effect.forkScoped (scope-owned).

   So the concrete leak path from the comments is not reproducible through today’s plugin API. The broader “long-lived idle V2 server allocation loop” report may still be real via another path — please re-open investigation if it still occurs on a recent binary, ideally with subscriber counts on pubsub.all / pubsub.typed vs live locations as dondetir suggested.
7. Just-Silver commented on Sep 25, 2026

   

   Just-Silver

   on Sep 25, 2026

   More actions

   Repro: plugin setup cleanup is never called when location.reload happens while the location is active (refCount > 0) — the old activation leaks

   This looks like a separate leak path from the one #38825 addressed (promise-plugin for await not calling return()). On v2 (v2.0.15) ctx.event.subscribe still exists, so that path was not the cause here. The maintainer note on #38825 says "no reachable repro of subscriber accumulation across location invalidate/expiry" — this is one.

   Root cause

   LocationServiceMap.reload() (packages/core/src/location-service-map.ts) calls locations.invalidate(ref) for every location. The custom invalidate in location-services.ts runs inner.invalidate(key).pipe(Effect.andThen(build?.close)), where inner is a LayerMap with idleTimeToLive: Duration.infinity. Effect's RcMap.invalidate:

   const entry = o.value
   MutableHashMap.remove(self.state.map, key)
   if (entry.refCount > 0) return          // only detaches the key, does NOT close the scope
   yield* core.scopeClose(entry.scope, core.exitVoid)

   So when a location still has an outstanding borrower (e.g. a session turn is running), the old Instance layer scope is not closed. build.close (LocationLifecycle.shutdown) detaches routing and interrupts, but the Layer scope — and therefore the plugin activation's Effect.acquireRelease(setup, cleanup) — is only released when refCount reaches 0. If reload replaces the entry, the old scope is orphaned and cleanup is never invoked.

   Minimal repro (trivial promise plugin)

   A plugin whose setup only logs and returns a cleanup that aborts its event subscription:

   export default {
     id: "leak-probe",
     async setup(ctx) {
       const tag = Math.random().toString(36).slice(2, 8)
       appendFileSync(LOG, `setup ${tag} loc=${ctx.location.directory}\n`)
       const abort = new AbortController()
       void (async () => {
         for await (const ev of ctx.event.subscribe({ signal: abort.signal })) { /* log */ }
       })()
       return () => { appendFileSync(LOG, `cleanup ${tag}\n`); abort.abort() }
     },
   }

   1. Send a prompt that runs ~10s in location D (keeps D borrowed).
   2. While it is running, run opencode reload.
   3. Observe the log:

   cleanup 99iyvj loc=C:\Users\...          ← idle location: closed
   cleanup x0yvc0 loc=E:\...                ← idle location: closed
   setup   l0tpb9 loc=D:\...                ← D: new activation
                                            ← NO `cleanup e40c0o loc=D` (the old D activation)!
   

   The old D activation (e40c0o) is still alive and keeps receiving session.execution.* from the global event stream after the reload. Idle locations are cleaned up correctly; only the active one leaks.

   I confirmed the inverse as well: when the cleanup is called, abort.abort() stops the loop (0 further events); when a probe omits abort(), the old loop keeps receiving events. So the plugin-side teardown is fine — the host simply never calls it in this case.

   Impact

   • A leaked activation subscribes to the global /api/event stream and passes the same ownership filter, so one session.execution.succeeded gets delivered N times → N× auto-continuation. Measured ratio (auto-continue receipts / succeeded) went from 1 to 3–4 after a few reloads while a turn was in flight.
   • The leaked closure retains ctx / location services, so memory only grows — same class of accumulation as this issue.

   Environment

   opencode v2.0.15 (v2 branch), Windows, reproduced over the HTTP API with opencode serve.

   The ask: LocationServiceMap.reload() should force-close (or wait for) the previous location scope even when it still has borrowers, instead of orphaning it. Happy to share the full probe script and logs.
8. underdark456 commented on Oct 1, 2026

   

   underdark456

   on Oct 1, 2026

   Last edited by underdark456

   More actions

   Hit the same thing @Just-Silver describes, on 2.0.20 with opencode serve --service and the desktop app on macOS. For me it shows up as leaked MCP processes. I never got the CPU loop.

   After about a day I had 692 processes under the service and ~39 full sets of my 12 MCP servers. Every local MCP gets spawned with cwd = location directory, so I grouped the processes by cwd. Most dirs had one set. The dir I actually work in had 8.

   Lining that up with the log:

   • idle expiry is fine. On location services evicted the old set gets killed, and the app reboots the location within ~2s.
   • location services reloaded count=31 leaks. It fires every time I edit opencode.json. Over the day there were 8 of those reloads and my working dir picked up 7 extra sets. Every surviving set started right before one of them. My guess is the location was still in use each time.

   Same RcMap.invalidate path as above. With refCount > 0 the old scope never closes, and the stdio MCP clients live in that scope too.

   The part that surprised me is that the orphaned graphs aren't inert. I killed all the stale MCP processes, then changed a single timeout field on one MCP server. This time there was no location services reloaded, only that server reconnecting per location. 9 fresh duplicates of it showed up in exactly the dirs where I had killed orphans. So the detached graphs keep watching config and respawning their MCP clients until the server restarts.

   Side effects for me were up to 74 duplicate kubectl port-forwards from a k8s MCP, and a screenshot MCP daemon whose session registry filled up and stopped working.

   Current workaround is a launchd job that dedupes MCP children of the service every 15 min, keyed by cwd plus argv and env. Env matters because two of my servers run the same binary with different env. That cleans up the processes, but the graphs stay in memory until opencode service restart.

   +1 to the ask: reload() should close the previous location scope, or interrupt its borrowers like the idle sweep does, instead of orphaning it.