Skip to content

[Bug]: Massive memory leak (60GB+ OOM crash) on Server #17908

Closed
Closed
[Bug]: Massive memory leak (60GB+ OOM crash) on Server#17908
Assignees
rekram1-node
Labels
bugSomething isn't workingcoreAnything pertaining to core functionality of the application (opencode server stuff)

Description

@breakstring
breakstring
opened on Mar 17, 2026

Description

I am running opencode serve on an Ubuntu server via tmux and connecting to it using the OpenCode Desktop app from another machine.

Everything works fine during normal usage. However, whenever I exit/close the Desktop app, the opencode server process experiences a massive memory leak, consuming over 60GB of RAM within seconds/minutes until it is killed by the Linux OOM Killer.

I have never encountered this issue in previous versions. It only started occurring frequently today right after I updated to v1.2.27.

Plugins

No response

OpenCode version

1.2.27

Steps to reproduce

  1. Start opencode serve on a remote Ubuntu server (inside a tmux session).
  2. Connect to the remote server using the OpenCode Desktop client.
  3. Use the agent normally (e.g., let it load context or run a task).
  4. Close/quit the OpenCode Desktop app.
  5. Check the server: the RAM spikes to 60GB+ and the opencode process is OOM killed.

Screenshot and/or share link

Logs (dmesg / syslog)

Here is the dmesg output showing the MainThread consuming ~60GB (anon-rss:62541124kB) before being killed:

❯ sudo dmesg -T | grep -i oom
[Tue Mar 17 03:55:36 2026] JITWorker invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Tue Mar 17 03:55:36 2026]  oom_kill_process+0x118/0x280
[Tue Mar 17 03:55:36 2026]  ? oom_evaluate_task+0x143/0x1e0
[Tue Mar 17 03:55:36 2026]  __alloc_pages_may_oom+0x10c/0x1d0
[Tue Mar 17 03:55:36 2026] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[Tue Mar 17 03:55:36 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/tmux-spawn-ed03c171-2996-4726-a65e-be3d7da9b485.scope,task=MainThread,pid=2147454,uid=1000
[Tue Mar 17 03:55:36 2026] Out of memory: Killed process 2147454 (MainThread) total-vm:91069720kB, anon-rss:62541124kB, file-rss:3200kB, shmem-rss:0kB, UID:1000 pgtables:133020kB oom_score_adj:0
[Tue Mar 17 03:55:40 2026] oom_reaper: reaped process 2147454 (MainThread), now anon-rss:576kB, file-rss:512kB, shmem-rss:0kB
[Tue Mar 17 04:04:04 2026] actix-rt|system invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Tue Mar 17 04:04:04 2026]  oom_kill_process+0x118/0x280
[Tue Mar 17 04:04:04 2026]  ? oom_evaluate_task+0x143/0x1e0
[Tue Mar 17 04:04:04 2026]  __alloc_pages_may_oom+0x10c/0x1d0
[Tue Mar 17 04:04:04 2026] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[Tue Mar 17 04:04:04 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-2a6bc6893a4742684583c1e4809b9de1bf5bdd4323f1891e1de1c978540ccd0c.scope,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/tmux-spawn-2ada0fe5-793a-4e9d-87d8-dbb338f7b991.scope,task=MainThread,pid=2159921,uid=1000
[Tue Mar 17 04:04:04 2026] Out of memory: Killed process 2159921 (MainThread) total-vm:99247288kB, anon-rss:63662424kB, file-rss:3680kB, shmem-rss:0kB, UID:1000 pgtables:149304kB oom_score_adj:0
[Tue Mar 17 04:04:08 2026] oom_reaper: reaped process 2159921 (MainThread), now anon-rss:0kB, file-rss:1108kB, shmem-rss:0kB
[Tue Mar 17 04:29:01 2026] dockerd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-500
[Tue Mar 17 04:29:01 2026]  oom_kill_process+0x118/0x280
[Tue Mar 17 04:29:01 2026]  ? oom_evaluate_task+0x143/0x1e0
[Tue Mar 17 04:29:01 2026]  __alloc_pages_may_oom+0x10c/0x1d0
[Tue Mar 17 04:29:01 2026] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[Tue Mar 17 04:29:01 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/tmux-spawn-8684de07-f0a7-49f6-8a86-aa5d99338712.scope,task=MainThread,pid=2194331,uid=1000
[Tue Mar 17 04:29:01 2026] Out of memory: Killed process 2194331 (MainThread) total-vm:98901612kB, anon-rss:64291072kB, file-rss:3520kB, shmem-rss:0kB, UID:1000 pgtables:148612kB oom_score_adj:0
[Tue Mar 17 04:29:05 2026] oom_reaper: reaped process 2194331 (MainThread), now anon-rss:0kB, file-rss:1152kB, shmem-rss:0kB

Operating System

Ubuntu 24.04

Terminal

No response

Activity

  1. breakstring
    added
    bugSomething isn't working
    on Mar 17, 2026
  2. github-actions
    assigned
    rekram1-node
    on Mar 17, 2026
  3. github-actions
    added
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on Mar 17, 2026

  4. github-actions commented on Mar 17, 2026

    @github-actions
    github-actionsBot
    on Mar 17, 2026 – with GitHub Actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

    If these describe the same root cause, it may be worth consolidating.

  5. breakstring
    changed the title [-][Bug]: Massive memory leak (60GB+ OOM crash) on Server when Desktop client disconnects[/-] [+][Bug]: Massive memory leak (60GB+ OOM crash) on Server[/+] on Mar 17, 2026

  6. breakstring commented on Mar 17, 2026

    @breakstring
    breakstring
    on Mar 17, 2026
    Author

    Update:
    After further testing, I realized the OOM crash is not strictly tied to disconnecting the client.

    I just experienced another OOM crash after about 15 minutes of usage while the Desktop client was still connected. The memory usage seems to grow rapidly even during active sessions.

    What's strange is that it was working normally for a few hours earlier this morning, but now it crashes consistently every 10-20 minutes. This suggests the leak might be triggered by specific background tasks (like indexing or SSE stream accumulation) that have become unstable in v1.2.27.

  7. breakstring commented on Mar 17, 2026

    @breakstring
    breakstring
    on Mar 17, 2026
    Author

    Crucial Update: > I've confirmed this OOM issue also occurs in TUI mode.

    I ran two separate opencode TUI sessions in two tmux windows for different projects. One session crashed with OOM (60GB+ RAM), while the other stayed perfectly normal.

    This strongly suggests the bug is related to session data handling (as discussed in #17837) rather than just a frontend connection leak. It seems when a specific session contains large tool outputs or long histories, the server/TUI process fails to handle the payload and triggers a massive memory spike.

  8. abrekhov commented on Mar 20, 2026

    @abrekhov
    abrekhov
    on Mar 20, 2026

    +1 — experiencing the same issue with opencode serve v1.2.24 running behind an auth reverse proxy (HTTP/2 → HTTP/1.1 on localhost:4096).

    Environment

    • 12 active workspaces (worktrees/sandboxes)
    • ~20 internal TCP connections from the proxy to port 4096
    • Linux, 125GB RAM

    Observations

    • RSS grows at ~80MB/min steadily, reaching 24GB+ after <2h
    • smaps_rollup shows 23.3GB of Pss_Anon (pure anonymous heap), all Private_Dirty
    • TUI mode does not exhibit the leak — only serve mode
    • The leak occurs even when connecting directly via SSH port-forward (bypassing the proxy), confirming it's in opencode serve itself
    • TCP socket analysis shows rwnd_limited:100% on 17/20 internal connections — opencode serve is writing data faster than the proxy can drain it, causing send-queue buildup of ~37MB

    Root Cause Analysis

    Traced through the source at v1.2.24 (c6262f9d4). The primary leak is in the SSE endpoints:

    1. server.ts:534-560 — Bus.subscribeAll() + setInterval(heartbeat, 10s) cleanup is gated entirely on stream.onAbort(). When a reverse proxy holds TCP connections open (standard behavior), onAbort never fires. The subscription and interval live forever, writing into a ReadableStream queue that grows without bound.

    2. routes/global.ts:85-106 — same pattern with GlobalBus.on("event", handler). Since GlobalBus receives events from all 12 instances, this is the most active leak site.

    3. bus/index.ts:52-63 — Bus.publish() does Promise.all(pending) over all subscribers including stale async SSE callbacks. If a dead stream's writeSSE() stalls on backpressure, the entire event pipeline stalls.

    4. session/index.ts:764 — structuredClone(part) on every LLM token update sends the full accumulated text to all stale subscribers, creating quadratic allocation.

    5. share/share-next.ts:22-65 — 4 Bus.subscribe() calls per instance with return values discarded. In serve mode, Instance.disposeAll() is never called, so these subscriptions accumulate forever.

    Checked v1.2.27 (dev HEAD)

    The SSE endpoints were partially refactored — AsyncQueue now decouples bus events from stream writes (fixing the Bus.publish stalling), and a finally { stop() } block provides a secondary cleanup path. However, the fundamental issue of cleanup being gated on stream.onAbort() persists. If writeSSE() buffers without throwing, the AsyncQueue grows without bound. The structuredClone(part) and ShareNext subscription leaks are also still present.

    Suggested Fix

    Use c.req.raw.signal (AbortSignal) as a secondary cleanup trigger. Bun populates this signal when the underlying connection closes, even when stream.onAbort() doesn't fire:

    c.req.raw.signal.addEventListener("abort", stop)

    This single line added to each SSE endpoint would cover the proxy disconnect case.

  9. jnorthrup commented on Mar 21, 2026

    @jnorthrup
    jnorthrup
    on Mar 21, 2026

    🚨 URGENT: Cross-Project Memory Leak Crisis - Hardware Damage Risk

    We investigated memory leaks across WebKit-based runtimes (Bun, Claude Code, OpenCode) and found a critical pattern affecting all embedders.

    Hardware Risk Statement

    On Apple Silicon Macs:

    • Memory exhaustion → kernel panic → forced power cycle
    • SOC (System on Chip) is non-replaceable ($500-1500 hardware loss)
    • Sustained memory pressure can cause thermal damage

    On Linux servers:

    • OOM kills → data corruption from unclean shutdowns
    • Kernel soft lockups (356s+) → system death requiring hard reset
    • Database corruption from abrupt termination

    Our Investigation Findings

    Bun Runtime (WebKit JSC):

    • 4 unique JSC GC memory leaks identified
    • JSC::SlotVisitor::drainFromShared race condition (7,214 frames)
    • WTF::HashTable::removeIterator (7,214 frames)
    • WTF::Vector<>::operator= (7,220 frames)
    • JSC::ASTBuilder::createForOfLoop (713 frames)

    OpenCode (from your issues):

    Common Root Cause

    WebKit JavaScriptCore parallel GC marking phase:

    • Multiple threads drain shared work queues
    • Race condition in SlotVisitor::drainFromShared
    • Iterator invalidation during concurrent access
    • Objects marked but not properly tracked → leak

    Upstream References

    Issue Project Status
    WPEWebKit #1622 SlotVisitor::drain crash Open
    WebKit #200863 SlotVisitor::visitChildren crash Open
    LLVM #115992 LSAN false positives Open
    Bun #28343 JSC GC tracking Open

    Tool We Built

    We created test/asan_tracker.zig (760 lines) that:

    • Parses ASAN/LSAN logs automatically
    • Classifies leaks by component (JSC, WTF, Bun, System)
    • Auto-symbolicates with source locations
    • Exports JSON for CI monitoring

    Happy to share this tool with OpenCode maintainers.

    Urgency: Auto-Close Bot Pattern

    We observed issues #17837 and #17185 were closed by github-actions bot within 2 hours for "contributing guidelines" - despite containing:

    • Detailed technical analysis
    • Reproduction steps
    • Root cause identification
    • PR with fixes

    This pattern is dangerous for memory safety issues:

    1. High-quality bug reports dismissed before maintainer review
    2. Hardware damage risk ignored
    3. Cross-project coordination blocked

    Recommended Actions

    Immediate:

    1. ⚠️ Disable auto-close for perf and bug labels
    2. ⚠️ Extend contribution window from 2h to 48h for complex bugs
    3. ⚠️ Require maintainer review before closing memory issues

    Short-term:

    1. Merge PR fix: multiple memory leaks in long-running sessions #10913 (4 leak fixes)
    2. Add RSS monitoring to CI
    3. Implement graceful degradation on memory pressure

    Long-term:

    1. Coordinate with WebKit upstream on GC fixes
    2. Share ASAN tracker tool across embedders
    3. Establish memory safety triage process

    Cross-Project Coordination

    We're coordinating with:

    Memory leaks don't respect project boundaries - this is an ecosystem-wide crisis.

    Evidence:

    Please prioritize this over formatting requirements.

  10. jnorthrup commented on Mar 21, 2026

    @jnorthrup
    jnorthrup
    on Mar 21, 2026

    🚨 URGENT: Cross-Project Memory Leak Crisis - Hardware Damage Risk

    We investigated memory leaks across WebKit-based runtimes (Bun, Claude Code, OpenCode) and found a critical pattern affecting all embedders.

    Hardware Risk Statement

    On Apple Silicon Macs:

    • Memory exhaustion → kernel panic → forced power cycle
    • SOC (System on Chip) is non-replaceable ($500-1500 hardware loss)
    • Sustained memory pressure can cause thermal damage

    On Linux servers:

    • OOM kills → data corruption from unclean shutdowns
    • Kernel soft lockups (356s+) → system death requiring hard reset
    • Database corruption from abrupt termination

    Our Investigation Findings

    Bun Runtime (WebKit JSC):

    • 4 unique JSC GC memory leaks identified via ASAN tracker
    • JSC::SlotVisitor::drainFromShared race condition (7,214 frames)
    • WTF::HashTable::removeIterator (7,214 frames)
    • WTF::Vector<>::operator= (7,220 frames)
    • JSC::ASTBuilder::createForOfLoop (713 frames)

    OpenCode (from your issues):

    Common Root Cause

    WebKit JavaScriptCore parallel GC marking phase:

    • Multiple threads drain shared work queues
    • Race condition in SlotVisitor::drainFromShared
    • Iterator invalidation during concurrent access
    • Objects marked but not properly tracked → leak

    Upstream References

    Issue Project Status
    WPEWebKit #1622 SlotVisitor::drain crash Open
    WebKit #200863 SlotVisitor::visitChildren crash Open
    LLVM #115992 LSAN false positives Open

    Tool We Built

    We created an ASAN tracker (760 lines of Zig) that:

    • Parses ASAN/LSAN logs automatically
    • Classifies leaks by component (JSC, WTF, Bun, System)
    • Auto-symbolicates with source locations
    • Exports JSON for CI monitoring

    Happy to share this tool with OpenCode maintainers.

    Urgency: Auto-Close Bot Pattern

    We observed issues #17837 and #17185 were closed by github-actions bot within 2 hours for "contributing guidelines" - despite containing:

    • Detailed technical analysis
    • Reproduction steps
    • Root cause identification
    • PR with fixes

    This pattern is dangerous for memory safety issues:

    1. High-quality bug reports dismissed before maintainer review
    2. Hardware damage risk ignored
    3. Cross-project coordination blocked

    Recommended Actions

    Immediate:

    1. ⚠️ Disable auto-close for perf and bug labels
    2. ⚠️ Extend contribution window from 2h to 48h for complex bugs
    3. ⚠️ Require maintainer review before closing memory issues

    Short-term:

    1. Merge PR fix: multiple memory leaks in long-running sessions #10913 (4 leak fixes)
    2. Add RSS monitoring to CI
    3. Implement graceful degradation on memory pressure

    Long-term:

    1. Coordinate with WebKit upstream on GC fixes
    2. Share ASAN tracker tool across embedders
    3. Establish memory safety triage process

    Cross-Project Coordination

    We're coordinating with:

    Memory leaks don't respect project boundaries - this is an ecosystem-wide crisis.

    Evidence:

    Please prioritize this over formatting requirements.

  11. herbievine commented on Mar 25, 2026

    @herbievine
    herbievine
    on Mar 25, 2026

    +1 using OC via serve. I also see gradual memory increments as @abrekhov describes, but am more concerned about the rapid increase of memory as shown in this graph:

    Image

    I am however using version 1.1.51, so it might suggest such change was not from 1.2.27 but from an entirely different issue.

  12. jnorthrup commented on Mar 25, 2026

    @jnorthrup
    jnorthrup
    on Mar 25, 2026

    looks like bun and OC conspire together to launch and forget JSC instances and webkit ... filling up the hard-drive paralyzing the system and triggering the desktop watchdog. MACOS compressed swap makes these astronomical amounts of virtual heap and you won't feel it for a while.

  13. sgtsquiggs commented on Mar 27, 2026

    @sgtsquiggs
    sgtsquiggs
    on Mar 27, 2026

    Seeing this too, since updating from 1.3.0 to 1.3.3

    Downgrading back to 1.3.0 resolved this issue for me.

  14. silent-night-no-trace commented on Apr 2, 2026

    @silent-night-no-trace
    silent-night-no-trace
    on Apr 2, 2026

    I am using version 1.3.2 and am still encountering this issue. How can I resolve it?

  15. thdxr commented on Apr 2, 2026

    @thdxr
    thdxr
    on Apr 2, 2026
    Member

    Closing in favor of #20695 where we're collecting all memory-related problems. Please follow that issue for updates.

  16. thdxr
    closed this as completedon Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcoreAnything pertaining to core functionality of the application (opencode server stuff)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions