Skip to content

opencode run: concurrent sessions crash with SQLITE_BUSY (busy_timeout=0) #21215

Closed as not planned

Description

@marcusquinn

Description

When multiple opencode run sessions execute concurrently (e.g., a CI/CD pipeline dispatching 8 workers simultaneously), sessions die silently mid-turn. The model initiates an API call (step_start event), but 0 tokens are generated and the process exits with code 0 — no error message, no retry.

Root Cause

The OpenCode session database (~/.local/share/opencode/opencode.db) uses WAL journal mode but has PRAGMA busy_timeout = 0:

$ sqlite3 ~/.local/share/opencode/opencode.db "PRAGMA busy_timeout"
0

WAL allows concurrent readers but only one writer at a time. With busy_timeout=0, any write that hits a lock from another session fails immediately with SQLITE_BUSY instead of retrying. When 8 concurrent opencode run sessions all write session state after tool calls, lock contention is constant and every session eventually hits SQLITE_BUSY.

Reproduction

# Launch 8 concurrent opencode run sessions
for i in $(seq 1 8); do
  opencode run "Run: echo step1_$i && echo step2_$i && echo step3_$i && echo done_$i" &
done
wait
# Some/all sessions will exit with 0 but produce incomplete output

Evidence

  • 8 workers dispatched in 3 minutes, all died mid-turn at random points (step 0, 7, 17, 20, 37)
  • Both Anthropic (claude-sonnet-4.6) and OpenAI (gpt-5.3-codex) models affected — it's the DB, not the API
  • Single worker tests pass every time — no contention
  • DB stats: 17,550 sessions, 1.86M part rows, 11.4 GB — large DB amplifies lock hold times

Session DB entries show the failure clearly:

-- Last message started but never completed (0 tokens)
SELECT id, json_extract(data, '$.time.created'), json_extract(data, '$.time.completed'), json_extract(data, '$.tokens.total')
FROM message WHERE session_id = '<session>' ORDER BY rowid DESC LIMIT 1;
-- Result: created=1775489146949, completed=NULL, tokens=NULL

Suggested Fix

Set PRAGMA busy_timeout = 5000 (5 seconds) on database connections. This gives concurrent writers time to retry instead of failing immediately. SQLite's WAL mode handles this well — the retry loop is built into the SQLite library, no application-level retry needed.

// In the database connection setup:
db.Exec("PRAGMA busy_timeout = 5000")

Alternatively, use a per-session data directory for opencode run to eliminate contention entirely, similar to how --session already scopes conversation state.

Environment

  • OpenCode v1.3.17 (also tested on 1.3.15, 1.3.16)
  • macOS 15 (Darwin 25.4.0)
  • SQLite WAL mode, 11.4 GB database

Activity

  1. added
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on Apr 6, 2026
  2. github-actions commented on Apr 6, 2026

    @github-actions
    Contributor

    This issue might be a duplicate of existing issues. Please check:

  3. marcusquinn commented on Apr 6, 2026

    @marcusquinn
    Author

    Thanks for flagging the related issues. After reviewing:

    This issue can be treated as a duplicate of #15188 with additional evidence: 8 concurrent opencode run sessions, reproducible across both Anthropic and OpenAI providers, and the specific busy_timeout=0 finding. Happy to close this in favour of #15188 if preferred — the key new data point is that a PRAGMA busy_timeout = 5000 one-liner would resolve the immediate crash-on-contention behaviour while the sharding work in #20935 is developed.

  4. marcusquinn commented on Apr 6, 2026

    @marcusquinn
    Author

    Bisect result: v1.3.16 works, v1.3.17 hangs

    Tested headless opencode run with a trivial task (read a file, return the version string):

    Version Result
    1.3.15 Completes in ~10s, exit 0
    1.3.16 Completes in ~10s, exit 0
    1.3.17 Hangs until killed (60s timeout, or indefinitely in worker dispatch)

    Only 4 commits between v1.3.16 and v1.3.17. Two are server-side (zen: rate limiter changes), one is Windows-only TUI. The remaining client-side change is #20399 (Cloudflare provider env vars).

    Likely cause: unconditional dep.auth() in provider init

    The Cloudflare provider commit moved yield* dep.auth(input.id) before the env-var fast-bail:

    // v1.3.16 — fast bail, no auth call for users without Cloudflare
    const accountId = Env.get("CLOUDFLARE_ACCOUNT_ID")
    if (!accountId) return { autoload: false }
    
    // v1.3.17 — auth call runs for ALL users before env check
    const auth = yield* dep.auth(input.id)
    const accountId = Env.get("CLOUDFLARE_ACCOUNT_ID") || (auth?.type === "api" ? auth.metadata?.accountId : undefined)

    Both cloudflare-workers-ai and cloudflare-ai-gateway were also added to INTERNAL_PLUGINS, so they load for every user regardless of configuration. In headless mode (opencode run), if dep.auth() doesn't resolve cleanly for an unconfigured provider, it blocks provider initialization and the entire session hangs.

    Environment

    • macOS (Apple Silicon)
    • Headless mode via opencode run -m anthropic/claude-sonnet-4-20250514 "..."
    • No Cloudflare env vars set (CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_KEY, etc.)
    • Affects both Anthropic and OpenAI model providers (provider init blocks before model selection)

    Suggested fix

    Restore the env-var fast-bail before dep.auth(), or skip auth for providers not configured in the user's settings:

    // Check env and baseURL first — only call dep.auth() if there's reason to
    if (input.options?.baseURL) return { autoload: false }
    const accountId = Env.get("CLOUDFLARE_ACCOUNT_ID")
    if (!accountId) {
      const auth = yield* dep.auth(input.id)
      // ... check auth metadata
    }
  5. marcusquinn commented on Apr 6, 2026

    @marcusquinn
    Author

    Additional data point: 1.3.17 hangs with --pure flag too (which skips external plugins), confirming this is in OpenCode core — specifically the Cloudflare internal plugins added to INTERNAL_PLUGINS in #20399, which load regardless of --pure.

    Test:

    $ opencode run --pure -m anthropic/claude-sonnet-4-20250514 --dir ~/Git/aidevops "Reply with exactly: CANARY_OK"
    > Build+ · claude-sonnet-4-20250514
    # ← hangs here indefinitely, killed after 30s timeout
    

    Same test with 1.3.16: completes in ~10s with exit 0.

  6. marcusquinn commented on Apr 6, 2026

    @marcusquinn
    Author

    Correction: root cause is SQLite contention, not Cloudflare provider init

    Our earlier bisect was misleading. The actual root cause is concurrent headless opencode run instances sharing opencode.db with busy_timeout=0. When multiple workers write simultaneously, SQLITE_BUSY fires immediately and silently breaks the streaming connection handler — the worker hangs at step_start with zero API errors logged.

    Evidence:

    • v1.3.17 with --pure hung → we blamed Cloudflare internal plugins
    • v1.3.15 and v1.3.16 also hung with concurrent workers → not version-specific
    • v1.3.17 completes immediately when given its own DB via XDG_DATA_HOME isolation
    • Interactive sessions (single instance, no contention) never had the issue

    Our fix: set XDG_DATA_HOME per worker so each gets its own opencode.db, then merge session data back to the shared DB after completion.

    Suggestion for opencode: consider setting busy_timeout to a non-zero value (e.g. 5000ms) in the SQLite connection. This would let concurrent instances coexist on a shared DB without silent failures. The current busy_timeout=0 means any lock collision is an immediate SQLITE_BUSY error, which appears to be swallowed somewhere in the streaming pipeline.

    Apologies for the noise on the Cloudflare provider theory — the timing correlation was coincidental.

  7. marcusquinn commented on Apr 18, 2026

    @marcusquinn
    Author

    Still reproducing on v1.4.11 — and the live log contradicts the obvious "busy_timeout was 0" diagnosis.

    I filed this originally against opencode run (headless), but it's now surfacing in interactive TUI sessions too. Wanted to share fresh evidence in case it helps pin down the exact path.

    The smoking gun

    ERROR 2026-04-18T02:50:17 +5015ms service=session.processor sessionID=… messageID=… error=database is locked stack=SQLiteError: database is locked
    

    The +5015ms is the critical bit: busy_timeout = 5000ms is being honored on the failing connection — the writer just held the WAL lock for longer than 5s. A single ~/.local/share/opencode/log/2026-04-17T234221.log from my install has 28 of these in one session. So this is a writer-duration problem, not a "timeout not set" problem. That reframes the fix — bumping busy_timeout alone won't help if a large bash-tool output takes 8s to serialize into part.

    What a fresh CLI connection actually shows

    $ sqlite3 ~/.local/share/opencode/opencode.db "PRAGMA busy_timeout;"
    0
    

    I'd read this as "some code path opens without applying the default" — but the +5015ms log contradicts that, so it's more likely that PRAGMA busy_timeout is session-local (expected) and the failing connection did have 5000ms set. The 0 just reflects the CLI's own new connection, not opencode's runtime state. Leaving this here because if anyone else repros with sqlite3 they'll see the same 0 and draw the wrong conclusion.

    A single interactive TUI is already multi-writer

    $ lsof ~/.local/share/opencode/opencode.db | awk 'NR>1 {print $2}' | sort -u | wc -l
    11
    $ lsof ~/.local/share/opencode/opencode.db | wc -l
    24   # incl. header
    

    11 opencode PIDs, 23 FDs on one DB file. A single TUI holds multiple FDs on opencode.db concurrently, so even with only one "user-visible session", the app is competing against itself for the single WAL writer slot. Any 9-13 pane workload amplifies this. I suspect this is why the per-session-tree sharding idea in #20935 and PR #21579 would help even for solo users.

    Current DB profile (context for anyone repro'ing)

    • opencode.db: 1.3 GB (1,314,107,392 bytes)
    • message: 60,123 rows
    • part: 168,942 rows
    • Free pages: ~0% (VACUUM isn't the main win here — lock-hold time is)

    User-visible failure mode (the biggest quality-of-life ask)

    When a lock fires, the session halts with the error banner and there's no automatic retry. I have to reload the thread and sometimes lose the in-flight tool call. An exponential-backoff retry on SQLITE_BUSY in the write path (50ms, 100ms, 250ms, 500ms, 1s → give up) would make almost all of these invisible without changing the architecture. That would buy time while the larger sharding work in #20935/#21579 lands.

    Related upstream threads

    What I'm shipping on my side (mitigation only)

    Weekly wal_checkpoint(TRUNCATE) + conditional VACUUM + PRAGMA optimize via a maintenance routine, scheduled off-hours. Doesn't touch busy_timeout or retry — those can only be fixed in opencode itself. Code: marcusquinn/aidevops#19630 if useful as a reference for what external tooling can and can't do.

    Happy to supply a full log sample or run instrumented builds if it would help.


    aidevops.sh v3.8.68 plugin for OpenCode v1.4.11 with claude-opus-4-7 spent 3h 30m and 5,001 tokens on this as a headless worker. Overall, 11d 10h since this issue was created.

  8. assigned and unassigned on Apr 26, 2026
  9. removed
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on May 3, 2026
  10. github-actions commented on Jul 6, 2026

    @github-actions
    Contributor

    To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions