Skip to content

High memory usage and database bloat in long-running OpenCode instances #16729

Closed as not planned
Closed as not planned
High memory usage and database bloat in long-running OpenCode instances#16729
Assignees
thdxr
Labels
coreAnything pertaining to core functionality of the application (opencode server stuff)perfIndicates a performance issue or need for optimization

Description

@BYK
BYK
opened on Mar 9, 2026

Description

A long-running OpenCode server instance (2+ days) consumes 1.76 GB (602 MB RSS + 1.15 GB swap) on an 8 GB system. The SQLite database grows to 1.99 GB with no automatic cleanup — 1,706 sessions spanning 53 days, 274K parts (1.49 GB) never pruned.

Root causes identified:

  • No data retention — sessions are only soft-archived, never deleted
  • auto_vacuum = OFF — deleted data never reclaims disk space
  • No periodic WAL checkpoint — WAL file grows unbounded during long sessions
  • Compaction marks parts but keeps full data — dead weight in SQLite
  • Bash tool output accumulates unboundedly — verbose commands can grow to hundreds of MB in the output string
  • Edit tool Levenshtein uses O(n×m) matrix — can spike to ~400 MB for 10K char strings
  • FileTime per-session tracking never pruned — slow memory leak over instance lifetime
  • RPC pending map has no timeout — orphaned promises leak if workers die
  • LSP diagnostics Map grows monotonically — empty entries never deleted

Measured metrics:

Metric Value
Main process RSS 602 MB
Main process swap 1.15 GB
Peak RSS (VmHWM) 1.64 GB
SQLite DB size 1.99 GB
DB parts table 1.49 GB
Sessions 1,706 (never auto-deleted)
auto_vacuum OFF

Plugins

No plugins

OpenCode version

0.1.44 (dev build)

Steps to reproduce

  1. Run opencode serve on a machine with 8 GB RAM
  2. Use it actively for 2+ days with multiple worktrees
  3. Monitor RSS with ps aux or /proc/<pid>/status — memory grows monotonically
  4. Check opencode.db file size — grows without bound since archived sessions are never deleted

Screenshot and/or share link

N/A

Operating System

Ubuntu 22.04 (Linux x86_64, 8 GB RAM, 16 GB swap)

Terminal

N/A (headless server mode via opencode serve)

Activity

  1. github-actions
    assigned
    thdxr
    on Mar 9, 2026
  2. github-actions
    added
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    perfIndicates a performance issue or need for optimization
    on Mar 9, 2026

  3. github-actions commented on Mar 9, 2026

    @github-actions
    github-actionsBot
    on Mar 9, 2026 – with GitHub Actions
    Contributor

    This issue doesn't fully meet our contributing guidelines.

    What needs to be fixed:

    • The issue does not use one of the required templates (Bug Report, Feature Request, or Question). Please resubmit using the appropriate template.
    • If this is a Bug Report, the template requires: a Description field, steps to reproduce, OpenCode version, and OS/environment information. None of these are present.

    Please edit this issue to address the above within 2 hours, or it will be automatically closed.

    This issue might be a duplicate of existing issues. Please check:

    If you believe this was flagged incorrectly, please let a maintainer know.

  4. github-actions
    added
    needs:complianceThis means the issue will auto-close after 2 hours.
    on Mar 9, 2026

  5. binarydoubling commented on Mar 9, 2026

    @binarydoubling
    binarydoubling
    on Mar 9, 2026

    Went ahead and tagged you up, this needs more attention and there's a lot to be done. Had my poor MacBook crash way too many times.

  6. github-actions commented on Mar 9, 2026

    @github-actions
    github-actionsBot
    on Mar 9, 2026 – with GitHub Actions
    Contributor

    This issue has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window.

    Feel free to open a new issue that follows our issue templates.

  7. github-actions
    removed
    needs:complianceThis means the issue will auto-close after 2 hours.
    on Mar 9, 2026
  8. github-actions
    closed this as not plannedon Mar 9, 2026

  9. BYK commented on Mar 9, 2026

    @BYK
    BYK
    on Mar 9, 2026
    ContributorAuthor

    Reformatted to match the bug report template. Reopening.

  10. jnorthrup commented on Mar 21, 2026

    @jnorthrup
    jnorthrup
    on Mar 21, 2026

    Comprehensive Memory Analysis - Related Findings

    Your analysis of long-running OpenCode instances is extremely thorough. We found similar patterns in Bun runtime investigation.

    Confirmed Leak Sources (Matching Our Findings)

    Your identified sources align with our JSC GC investigation:

    • ✅ Bash tool output accumulates unboundedly → We found similar in shell tool buffers
    • ✅ LSP diagnostics Map grows monotonically → We found this in JSC GC HashMap
    • ✅ No periodic cleanup → We found GC marking phase never completes
    • ✅ SQLite never pruned → Similar to our session storage leaks

    Additional Context

    Hardware Risk: On Apple Silicon Macs, memory exhaustion leading to kernel panic → forced power cycle → potential SOC damage (non-replaceable, $500-1500 hardware loss).

    Our Investigation

    We built an ASAN tracker that identified 4 unique JSC GC leaks:

    • JSC::SlotVisitor::drainFromShared race
    • WTF::HashTable::removeIterator
    • WTF::Vector<>::operator=
    • JSC::ASTBuilder::createForOfLoop

    Cross-Project Issue

    This affects all WebKit-based runtimes. We're coordinating upstream:

    Your 8 root causes + our 4 JSC GC leaks = comprehensive picture of WebKit memory issues.

  11. jnorthrup commented on Mar 21, 2026

    @jnorthrup
    jnorthrup
    on Mar 21, 2026

    🚨 URGENT: Cross-Project Memory Leak Crisis - Hardware Damage Risk

    We investigated memory leaks across WebKit-based runtimes (Bun, Claude Code, OpenCode) and found a critical pattern affecting all embedders.

    Hardware Risk Statement

    On Apple Silicon Macs:

    • Memory exhaustion → kernel panic → forced power cycle
    • SOC (System on Chip) is non-replaceable ($500-1500 hardware loss)
    • Sustained memory pressure can cause thermal damage

    On Linux servers:

    • OOM kills → data corruption from unclean shutdowns
    • Kernel soft lockups (356s+) → system death requiring hard reset
    • Database corruption from abrupt termination

    Our Investigation Findings

    Bun Runtime (WebKit JSC):

    • 4 unique JSC GC memory leaks identified
    • JSC::SlotVisitor::drainFromShared race condition (7,214 frames)
    • WTF::HashTable::removeIterator (7,214 frames)
    • WTF::Vector<>::operator= (7,220 frames)
    • JSC::ASTBuilder::createForOfLoop (713 frames)

    OpenCode (from your issues):

    Common Root Cause

    WebKit JavaScriptCore parallel GC marking phase:

    • Multiple threads drain shared work queues
    • Race condition in SlotVisitor::drainFromShared
    • Iterator invalidation during concurrent access
    • Objects marked but not properly tracked → leak

    Upstream References

    Issue Project Status
    WPEWebKit #1622 SlotVisitor::drain crash Open
    WebKit #200863 SlotVisitor::visitChildren crash Open
    LLVM #115992 LSAN false positives Open
    Bun #28343 JSC GC tracking Open

    Tool We Built

    We created test/asan_tracker.zig (760 lines) that:

    • Parses ASAN/LSAN logs automatically
    • Classifies leaks by component (JSC, WTF, Bun, System)
    • Auto-symbolicates with source locations
    • Exports JSON for CI monitoring

    Happy to share this tool with OpenCode maintainers.

    Urgency: Auto-Close Bot Pattern

    We observed issues #17837 and #17185 were closed by github-actions bot within 2 hours for "contributing guidelines" - despite containing:

    • Detailed technical analysis
    • Reproduction steps
    • Root cause identification
    • PR with fixes

    This pattern is dangerous for memory safety issues:

    1. High-quality bug reports dismissed before maintainer review
    2. Hardware damage risk ignored
    3. Cross-project coordination blocked

    Recommended Actions

    Immediate:

    1. ⚠️ Disable auto-close for perf and bug labels
    2. ⚠️ Extend contribution window from 2h to 48h for complex bugs
    3. ⚠️ Require maintainer review before closing memory issues

    Short-term:

    1. Merge PR fix: multiple memory leaks in long-running sessions #10913 (4 leak fixes)
    2. Add RSS monitoring to CI
    3. Implement graceful degradation on memory pressure

    Long-term:

    1. Coordinate with WebKit upstream on GC fixes
    2. Share ASAN tracker tool across embedders
    3. Establish memory safety triage process

    Cross-Project Coordination

    We're coordinating with:

    Memory leaks don't respect project boundaries - this is an ecosystem-wide crisis.

    Evidence:

    Please prioritize this over formatting requirements.

  12. jnorthrup commented on Mar 21, 2026

    @jnorthrup
    jnorthrup
    on Mar 21, 2026

    🚨 URGENT: Cross-Project Memory Leak Crisis - Hardware Damage Risk

    We investigated memory leaks across WebKit-based runtimes (Bun, Claude Code, OpenCode) and found a critical pattern affecting all embedders.

    Hardware Risk Statement

    On Apple Silicon Macs:

    • Memory exhaustion → kernel panic → forced power cycle
    • SOC (System on Chip) is non-replaceable ($500-1500 hardware loss)
    • Sustained memory pressure can cause thermal damage

    On Linux servers:

    • OOM kills → data corruption from unclean shutdowns
    • Kernel soft lockups (356s+) → system death requiring hard reset
    • Database corruption from abrupt termination

    Our Investigation Findings

    Bun Runtime (WebKit JSC):

    • 4 unique JSC GC memory leaks identified via ASAN tracker
    • JSC::SlotVisitor::drainFromShared race condition (7,214 frames)
    • WTF::HashTable::removeIterator (7,214 frames)
    • WTF::Vector<>::operator= (7,220 frames)
    • JSC::ASTBuilder::createForOfLoop (713 frames)

    OpenCode (from your issues):

    Common Root Cause

    WebKit JavaScriptCore parallel GC marking phase:

    • Multiple threads drain shared work queues
    • Race condition in SlotVisitor::drainFromShared
    • Iterator invalidation during concurrent access
    • Objects marked but not properly tracked → leak

    Upstream References

    Issue Project Status
    WPEWebKit #1622 SlotVisitor::drain crash Open
    WebKit #200863 SlotVisitor::visitChildren crash Open
    LLVM #115992 LSAN false positives Open

    Tool We Built

    We created an ASAN tracker (760 lines of Zig) that:

    • Parses ASAN/LSAN logs automatically
    • Classifies leaks by component (JSC, WTF, Bun, System)
    • Auto-symbolicates with source locations
    • Exports JSON for CI monitoring

    Happy to share this tool with OpenCode maintainers.

    Urgency: Auto-Close Bot Pattern

    We observed issues #17837 and #17185 were closed by github-actions bot within 2 hours for "contributing guidelines" - despite containing:

    • Detailed technical analysis
    • Reproduction steps
    • Root cause identification
    • PR with fixes

    This pattern is dangerous for memory safety issues:

    1. High-quality bug reports dismissed before maintainer review
    2. Hardware damage risk ignored
    3. Cross-project coordination blocked

    Recommended Actions

    Immediate:

    1. ⚠️ Disable auto-close for perf and bug labels
    2. ⚠️ Extend contribution window from 2h to 48h for complex bugs
    3. ⚠️ Require maintainer review before closing memory issues

    Short-term:

    1. Merge PR fix: multiple memory leaks in long-running sessions #10913 (4 leak fixes)
    2. Add RSS monitoring to CI
    3. Implement graceful degradation on memory pressure

    Long-term:

    1. Coordinate with WebKit upstream on GC fixes
    2. Share ASAN tracker tool across embedders
    3. Establish memory safety triage process

    Cross-Project Coordination

    We're coordinating with:

    Memory leaks don't respect project boundaries - this is an ecosystem-wide crisis.

    Evidence:

    Please prioritize this over formatting requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

coreAnything pertaining to core functionality of the application (opencode server stuff)perfIndicates a performance issue or need for optimization

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions