Repository navigation
opencode web becomes unresponsive after hours: SSE connection leak causes 7GB+ memory growth and event loop freeze #17628
Description
Description
Running opencode web on a Linux server (Debian, 56 cores, 125GB RAM), the HTTP server becomes completely unresponsive after a variable period (sometimes minutes, sometimes days). The process remains alive but refuses all HTTP requests, including /global/health. PM2 does not detect the failure because the process never exits.
This is related to #15645, #14091, and #15149, but provides production-level diagnostic data showing the full failure cascade.
Root Cause Analysis
Three bugs in the SSE implementation (server.ts:520-554 and routes/global.ts:71-107) create a fatal chain reaction:
Bug 1: Heartbeat writes are fire-and-forget (server.ts:537-543)
const heartbeat = setInterval(() => {
stream.writeSSE({ // ❌ NOT awaited, no try/catch
data: JSON.stringify({ type: "server.heartbeat", properties: {} }),
})
}, 10_000)When a client disconnects but onAbort hasn't fired, this interval keeps writing to a dead stream every 10 seconds forever. Errors are silently swallowed.
Bug 2: Bus event writes to dead connections block all event publishing (server.ts:527-533)
const unsub = Bus.subscribeAll(async (event) => {
await stream.writeSSE({ data: JSON.stringify(event) }) // ❌ hangs on dead connection
})Bus.publish() calls Promise.all(pending) across all subscribers. If one subscriber's stream.writeSSE() hangs on a dead connection, it blocks event delivery to every other subscriber.
Bug 3: onAbort never fires for CLOSE-WAIT connections
When TCP enters CLOSE-WAIT (client sent FIN, server hasn't acknowledged), Hono/Bun's stream.onAbort() may never trigger. This means:
clearInterval(heartbeat)never runs → heartbeat interval leaksunsub()never runs → Bus subscription leaks- The Promise at line 546 never resolves → stream handler stays alive forever
Bonus: Proxy to app.opencode.ai has no timeout (server.ts:560)
const response = await proxy(`https://app.opencode.ai${path}`, { ... })
// ❌ No AbortSignal, no timeout — can hang indefinitelyProduction Diagnostic Data (captured during freeze)
Memory growth
| Metric | At startup | At freeze (~45 min) |
|---|---|---|
| Memory | 528 MB (0.4%) | 7,134 MB (5.4%) |
| CPU | 49% | 33% |
| TCP connections | 4 | 200+ |
| CLOSE-WAIT connections | 0 | 30+ |
TCP connection state (abridged)
# All ESTAB connections from port 4069 have Send-Q of 798-800 bytes (write buffer full)
ESTAB 799 0 127.0.0.1:4069 127.0.0.1:56298
ESTAB 799 0 127.0.0.1:4069 127.0.0.1:53070
ESTAB 799 0 127.0.0.1:4069 127.0.0.1:40474
...
# CLOSE-WAIT connections from external client — server never closed its side
CLOSE-WAIT 593 0 38.97.xx.5:4069 13.193.97.167:56050
CLOSE-WAIT 593 0 38.97.xx.5:4069 13.193.97.167:63601
CLOSE-WAIT 593 0 38.97.xx.5:4069 13.193.97.167:63574
... (30+ more)
Key observations
curl -m 5 http://localhost:4069/global/health→ timeout (event loop frozen)- Process still alive in
ps aux(PID exists, 33% CPU, 7.1GB RSS) dmesgshows no OOM kill- Application log ends cleanly with normal health check responses — no error, no crash
Steps to Reproduce
- Run
opencode web --port 4069 --hostname 0.0.0.0on a Linux server - Access the web UI from a browser and use it normally
- Occasionally close/reopen browser tabs (creates SSE reconnections)
- Wait (variable: minutes to days)
- HTTP server becomes completely unresponsive while process stays alive
Suggested Fix
// 1. Add try/catch + await to heartbeat writes
const heartbeat = setInterval(async () => {
try {
await stream.writeSSE({ data: JSON.stringify({ type: "server.heartbeat", properties: {} }) })
} catch {
clearInterval(heartbeat)
unsub()
stream.close()
}
}, 10_000)
// 2. Add try/catch to bus event writes
const unsub = Bus.subscribeAll(async (event) => {
try {
await stream.writeSSE({ data: JSON.stringify(event) })
} catch {
clearInterval(heartbeat)
unsub()
stream.close()
}
})
// 3. Add timeout to proxy
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 10_000)
const response = await proxy(`https://app.opencode.ai${path}`, {
...c.req,
signal: controller.signal,
headers: { ...c.req.raw.headers, host: "app.opencode.ai" },
})
clearTimeout(timeout)Environment
- OS: Debian 12 (6.1.0-43-amd64)
- Server: 56 cores, 125GB RAM
- OpenCode version: Latest from npm (
opencode-ai) - Node: v24.12.0
- Runtime: Bun (embedded in opencode)
- Setup:
nohup opencode web/ PM2 managed, accessed via Nginx reverse proxy
Activity
github-actions commentedon Mar 15, 2026
This issue might be a duplicate of existing issues. Please check:
writeSSEand Bus subscription leakserveandwebcommands leak child processes on exit; SSE heartbeat interval leaks on server-initiated close #14091: SSE heartbeat interval leaks on server-initiated close — covers the same heartbeatclearIntervalnever running andunsub()never firingThese issues appear to describe the same root cause and failure mode. The maintainers may want to consolidate or track this as additional production evidence on one of those existing issues.
jnorthrup commentedon Mar 21, 2026
🚨 URGENT: Cross-Project Memory Leak Crisis - Hardware Damage Risk
We investigated memory leaks across WebKit-based runtimes (Bun, Claude Code, OpenCode) and found a critical pattern affecting all embedders.
Hardware Risk Statement
On Apple Silicon Macs:
On Linux servers:
Our Investigation Findings
Bun Runtime (WebKit JSC):
JSC::SlotVisitor::drainFromSharedrace condition (7,214 frames)WTF::HashTable::removeIterator(7,214 frames)WTF::Vector<>::operator=(7,220 frames)JSC::ASTBuilder::createForOfLoop(713 frames)OpenCode (from your issues):
Common Root Cause
WebKit JavaScriptCore parallel GC marking phase:
SlotVisitor::drainFromSharedUpstream References
Tool We Built
We created
test/asan_tracker.zig(760 lines) that:Happy to share this tool with OpenCode maintainers.
Urgency: Auto-Close Bot Pattern
We observed issues #17837 and #17185 were closed by github-actions bot within 2 hours for "contributing guidelines" - despite containing:
This pattern is dangerous for memory safety issues:
Recommended Actions
Immediate:
perfandbuglabelsShort-term:
Long-term:
Cross-Project Coordination
We're coordinating with:
Memory leaks don't respect project boundaries - this is an ecosystem-wide crisis.
Evidence:
Please prioritize this over formatting requirements.
jnorthrup commentedon Mar 21, 2026
🚨 URGENT: Cross-Project Memory Leak Crisis - Hardware Damage Risk
We investigated memory leaks across WebKit-based runtimes (Bun, Claude Code, OpenCode) and found a critical pattern affecting all embedders.
Hardware Risk Statement
On Apple Silicon Macs:
On Linux servers:
Our Investigation Findings
Bun Runtime (WebKit JSC):
JSC::SlotVisitor::drainFromSharedrace condition (7,214 frames)WTF::HashTable::removeIterator(7,214 frames)WTF::Vector<>::operator=(7,220 frames)JSC::ASTBuilder::createForOfLoop(713 frames)OpenCode (from your issues):
Common Root Cause
WebKit JavaScriptCore parallel GC marking phase:
SlotVisitor::drainFromSharedUpstream References
Tool We Built
We created an ASAN tracker (760 lines of Zig) that:
Happy to share this tool with OpenCode maintainers.
Urgency: Auto-Close Bot Pattern
We observed issues #17837 and #17185 were closed by github-actions bot within 2 hours for "contributing guidelines" - despite containing:
This pattern is dangerous for memory safety issues:
Recommended Actions
Immediate:
perfandbuglabelsShort-term:
Long-term:
Cross-Project Coordination
We're coordinating with:
Memory leaks don't respect project boundaries - this is an ecosystem-wide crisis.
Evidence:
Please prioritize this over formatting requirements.
thdxr commentedon Apr 2, 2026
Closing in favor of #20695 where we're collecting all memory-related problems. Please follow that issue for updates.
feat(b.core): out-of-process svc-opencode-watchdog for wedge recovery (…