Skip to content

SQLite database corruption (database disk image is malformed) when running concurrent sessions on NFS #14970

Closed as not planned

Description

@jerry-xu0514

Bug Report

OpenCode version: latest (as of 2026-02-24)
OS: Linux 6.6.105+ (NFS-mounted home directory)
Runtime: Bun

Description

Opening multiple opencode sessions in the same repository corrupts the shared SQLite database (~/.local/share/opencode/opencode.db), making opencode unable to start until the database is manually deleted.

Error

{
  "name": "UnknownError",
  "data": {
    "message": "SQLiteError: database disk image is malformed\n    at values (unknown)\n    at get (../../node_modules/.bun/drizzle-orm@1.0.0-beta.12-a5629fb+1f516382cd87cd66/node_modules/drizzle-orm/bun-sqlite/session.js:91:25)\n    at run (node:async_hooks:62:22)\n    at use (src/storage/db.ts:111:28)\n    at fromDirectory (src/project/project.ts:193:26)\n    at async <anonymous> (src/project/instance.ts:27:52)\n    at async provide (src/project/instance.ts:40:23)\n    at processTicksAndRejections (native:7:39)"
  }
}

Steps to Reproduce

  1. Have opencode installed with home directory on NFS
  2. Open an opencode session in a repository
  3. Open a second opencode session in the same (or different) repository
  4. Send a message in either session
  5. SQLite database corrupts almost immediately — opencode crashes with the above error
  6. All subsequent attempts to start opencode also crash until the database is manually deleted

Root Cause (Suspected)

All opencode sessions share a single SQLite database (~/.local/share/opencode/opencode.db) using WAL mode. Concurrent writers likely cause corruption due to:

  1. NFS + SQLite locking: SQLite relies on POSIX fcntl() advisory locking, which is notoriously unreliable on NFS. The WAL -shm file uses shared memory mappings that don't work correctly over network filesystems.
  2. bun-sqlite concurrency: Even on local filesystems, multiple processes writing to the same bun-sqlite database without explicit coordination (e.g., PRAGMA busy_timeout, retry logic, or a single-writer architecture) can corrupt the database.

Evidence of NFS involvement: stale .nfs* handle files were present alongside the corrupted database in ~/.local/share/opencode/.

Workaround

Delete the corrupted database and let opencode recreate it (loses all session history):

rm ~/.local/share/opencode/opencode.db ~/.local/share/opencode/opencode.db-shm ~/.local/share/opencode/opencode.db-wal

Suggested Fixes

  • Set PRAGMA busy_timeout to handle concurrent access gracefully instead of corrupting
  • Use PRAGMA journal_mode=DELETE instead of WAL when on a network filesystem (or detect NFS and warn)
  • Consider per-project databases instead of a single global one
  • Add integrity checks on startup with a graceful recovery path instead of crashing

Related Issues

Activity

  1. added
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on Feb 24, 2026
  2. jerry-xu0514 commented on Feb 25, 2026

    @jerry-xu0514
    Author

    Follow-up Investigation (2026-02-25) — Third Corruption Incident

    The corruption has occurred again today (Feb 25), the third time in one week. Sessions were running in separate repositories this time. Posting detailed findings with clear separation of observed facts vs. inferences.


    Observed Facts

    1. The database is on NFS — confirmed

    $ df -T ~/.local/share/opencode/
    Filesystem                         Type  ...  Mounted on
    10.125.224.2://shared_home/jerryxu nfs   ...  /mnt/home
    
    $ stat --file-system ~/.local/share/opencode/
      Type: nfs
    
    $ python3 -c "... statfs() ..."
      f_type: 0x6969 (NFS magic number)
    

    NFS mount options: vers=3, local_lock=none, proto=tcp, nconnect=7, hard

    2. The current database is corrupted right now

    $ PRAGMA integrity_check
    *** in database main ***
    Tree 12 page 12: unable to get the page. error code=522
    Tree 6 page 59: unable to get the page. error code=522
    

    Database stats: 122 pages × 4096 bytes = 500KB main db, WAL = 2.7MB (656 frames), SHM = 32KB.

    The database was recently recreated after the previous corruption (only 2 sessions, 18 messages, 71 parts exist) — meaning it corrupted again almost immediately after recovery.

    3. Six stale .nfs* handle files — physical evidence of the corruption mechanism

    ~/.local/share/opencode/
      .nfs...7700000062  45MB   Feb 18 16:36  (corrupted db, incident #1)
      .nfs...7800000064  5.1MB  Feb 18 18:32  (its WAL file)
      .nfs...7900000063  32KB   Feb 18 18:32  (its SHM file)
    
      .nfs...0200000091  191MB  Feb 24 15:25  (corrupted db, incident #2)
      .nfs...5600000093  5.7MB  Feb 24 15:33  (its WAL file)
      .nfs...5700000092  32KB   Feb 24 15:33  (its SHM file)
    

    These .nfs* files are NFS "silly rename" artifacts. They are created when a file is deleted (rm) while another process still has it open. NFS cannot truly delete the file, so it renames it to .nfsXXXX and deletes it when the last file handle closes.

    This means: when the corrupted database was deleted to recover, at least one other opencode process still had the database file open. This confirms concurrent multi-process access was occurring.

    Note the database sizes grew dramatically between incidents: 45MB → 191MB, which is consistent with unchecked WAL growth from uncoordinated concurrent writers.

    4. Installed opencode v1.2.14 has no NFS protection

    $ opencode --version
    1.2.14
    
    $ strings $(which opencode) | grep -c "resolveDbDir"
    0
    
    $ strings $(which opencode) | grep -c "isNFS"
    0
    

    The installed binary does not contain any NFS detection, integrity check, or fallback logic.

    5. PRAGMA busy_timeout IS set in the source code

    The original issue suggested adding busy_timeout — it's actually already there (PRAGMA busy_timeout = 5000). This pragma is not sufficient to prevent corruption on NFS because the problem is not lock contention timeout but rather incoherent shared memory (the -shm mmap). busy_timeout only helps when locks work correctly but are held by another process; on NFS, the locks themselves are unreliable.

    6. WAL checkpoint behavior reveals the corruption mechanism

    On first connection via python3, integrity check failed (errors above). On second connection moments later, integrity check passed and all tables were readable. This is consistent with WAL-index (-shm) corruption rather than main database file corruption: the second connection rebuilt the WAL index from scratch, recovering the data. This is exactly the failure mode described in SQLite's documentation on WAL mode over network filesystems — the -shm shared memory mappings become incoherent across NFS clients.


    Inferences (not directly observed, but strongly supported)

    1. The corruption is caused by WAL mode on NFS, not by a concurrency bug in opencode's code. The .nfs* stale handles, the -shm file corruption pattern, and the NFS mount type all point to this. SQLite's own documentation explicitly warns against this configuration.

    2. Sessions on different repos still collide because all projects share one global database file (~/.local/share/opencode/opencode.db). Even if the sessions are logically independent, they write to the same SQLite file and contend for the same WAL/SHM.

    3. The corruption can happen with a single session on NFS if the process crashes or is killed without cleanly closing the database, leaving the WAL in an inconsistent state that the next process cannot recover from over NFS. (Though multi-session definitely makes it more likely.)


    Assessment of Proposed Fixes

    We've prototyped fixes locally (not yet released). Here's what they do and whether they'd work:

    Fix 1: NFS detection + /tmp fallback

    function isNFS(dir: string): boolean {
      return statfsSync(dir).type === 0x6969
    }
    
    function resolveDbDir(): string {
      if (!isNFS(Global.Path.data)) return Global.Path.data
      const localDir = path.join(os.tmpdir(), "opencode-" + os.userInfo().uid)
      mkdirSync(localDir, { recursive: true })
      return localDir
    }

    Would this fix the corruption? Yes. We confirmed that statfsSync().type returns 0x6969 on this system, so the detection works correctly. Moving the database to /tmp avoids NFS entirely, and WAL mode works perfectly on local filesystems.

    Trade-off: /tmp is ephemeral — session history is lost on machine reboot. For our use case (ephemeral dev containers), this is acceptable. For persistent workstations with NFS home dirs, a configurable local path (e.g., OPENCODE_DATA_DIR env var) would be better.

    Fix 2: Integrity check + auto-recreation on startup

    const result = sqlite.prepare("PRAGMA integrity_check").get()
    if (result?.integrity_check !== "ok") {
      sqlite.close()
      removeDatabase(dbPath)
      sqlite = new BunDatabase(dbPath, { create: true })
    }

    Would this prevent the hard crash? Yes. Instead of crashing with "database disk image is malformed", opencode would detect the corruption, log a warning, delete the database, and start fresh. This is defense-in-depth — it doesn't prevent corruption but makes it self-healing.

    Would this prevent data loss? No. The database is still recreated from scratch; all session history is lost. But at least the user doesn't have to manually intervene.

    Fix 3 (not implemented): PRAGMA journal_mode = DELETE on NFS

    An alternative to the /tmp fallback. DELETE mode doesn't use shared memory mappings, so it's more NFS-compatible. However, it's slower for concurrent reads and still relies on NFS advisory locks, which are fragile. We chose the /tmp approach as more robust.


    Recommendation

    Ship fixes 1 + 2 together:

    • NFS detection + local fallback prevents the root cause
    • Integrity check provides resilience even on local filesystems (e.g., power loss, OOM kills)
    • Consider adding an OPENCODE_DB_PATH or OPENCODE_DATA_DIR env var override for users who want control over where the database lives

    Timeline

    • Feb 18: First corruption, manually deleted db to recover
    • Feb 24: Second corruption, manually deleted db to recover
    • Feb 25: Third corruption (current), sessions on separate repos
  3. added a commit that references this issue on Feb 25, 2026
    8ac922f
  4. Dulani commented on Feb 27, 2026

    @Dulani

    We’re seeing a very similar failure mode, but triggered by multi-server access to a shared NFS home (not just multiple sessions on one host).

    Setup:

    • NFS-mounted home directory shared across multiple servers.
    • opencode uses a single DB at ~/.local/share/opencode/opencode.db.
    • We started opencode on server A, then later on server B (same user, same NFS home). That cross-host access corrupted the SQLite DB with "database disk image is malformed."

    Why this matters:

    • Even if sessions are in different repos, they still write to the same DB. When the same home is mounted on multiple servers, WAL + SHM behave poorly across NFS clients.

    Workaround that solved it for us:

    • Make XDG_DATA_HOME host-specific (set in .bashrc), e.g.
      export XDG_DATA_HOME="/home/RAND.ORG/dwoods/.local/share/opencode-vincent"
    • This keeps opencode.db per-server and avoids NFS contention/cross-host WAL/SHM corruption.
  5. Miskamyasa commented on Mar 11, 2026

    @Miskamyasa

    About workaround. It means, you need to auth each instance separatly or provide keys to each container. There is no way to separate auth.json from data folder as I can see from the code.

  6. zcutlip commented on Mar 23, 2026

    @zcutlip

    I'm using opencode in a sandboxed container, and this happened. I asked opencode running from my host to try and repair the container instance. May not work every time, but it worked in this case, so I had opencode write a skill for what it did. Attached in case it's useful. Adapt paths accordingly.

    Prerequisite: you need a functioning opencode instance independent of the instance with the corrupted database.

    ---
    name: sqlite-recover
    description: Recover corrupted SQLite databases
    ---
    
    ## When to Use
    
    - User reports SQLite errors like "database disk image is malformed"
    - Error code SQLITE_CORRUPT appears in stack traces
    - `PRAGMA integrity_check` returns errors
    
    ## Recovery Process
    
    ### 1. Locate the Database
    
    Common locations for opencode in contai:
    - `~/.local/share/contai/home/.local/share/opencode/opencode.db`
    - `~/.local/share/opencode/opencode.db`
    
    ### 2. Verify Corruption
    
    ```sh
    sqlite3 <database_path> "PRAGMA integrity_check;"
    ```
    
    If corrupted, this will return errors instead of "ok".
    
    ### 3. Create Recovery SQL
    
    ```sh
    cd <database_directory>
    sqlite3 <database_name> ".recover" > recovery.sql
    ```
    
    ### 4. Recreate Database
    
    ```sh
    # Backup corrupt files
    mv <database_name> <database_name>.corrupt
    mv <database_name>-shm <database_name>-shm.corrupt 2>/dev/null || true
    mv <database_name>-wal <database_name>-wal.corrupt 2>/dev/null || true
    
    # Create new database from recovery
    sqlite3 <database_name> < recovery.sql
    ```
    
    ### 5. Verify Recovery
    
    ```sh
    sqlite3 <database_name> "PRAGMA integrity_check;"
    ```
    
    Should return "ok".
    
    ### 6. Check Recovered Data
    
    ```sh
    sqlite3 <database_name> "SELECT 'tables:', COUNT(*) FROM sqlite_master WHERE type='table';"
    ```
    
    ### 7. Cleanup
    
    ```sh
    rm *.corrupt recovery.sql
    ```
    
    ## Notes
    
    - `.recover` extracts all readable data, some data may be lost
    - WAL and SHM files are temporary and can be safely removed
    - Always verify integrity after recovery before deleting backups
  7. etherops commented on Mar 25, 2026

    @etherops

    Ran into the same issue running like this:
    docker run -ti --rm -v ~/.local/share/opencode:/root/.local/share/opencode opencode-dev

    Attempted out of the box sqllite .recover which restored session and part tables but the message table pages were on the corrupted blocks and came back empty - leaving me with historical session but ZERO conversation history! no good.

    Was able to start a new opencode database and Opus 4.6 figured out how to actually surigicall reconstruct messages from the parts table. Here is the script it came up with in case someone else needs this workaround. good luck we're all counting on you.

    https://gist.github.com/etherops/df78300a0ce6eeb08e056533bdfa30d9

    My suggestions if you run into this and .recover isn't good enough, blow away and get a fresh opencode database and give it that python code as guidance instead of trying to run this on your system which will break obv due to hard coded filenames etc.

    Notes:

    • Swap gotchas: don't replace opencode.db while opencode is running, and delete -shm/-wal alongside the swap.
    • if you manually reconstruct messages without including modelID, providerID, mode, agent, and path in the data JSON, opencode crashes on load with undefined is not an object (evaluating 'str3.replace').
  8. removed
    coreAnything pertaining to core functionality of the application (opencode server stuff)
    on May 3, 2026
  9. mattjohnsonpint commented on May 5, 2026

    @mattjohnsonpint

    I hit this multiple times a week using a Colima-based devcontainer (ie., virtio not NFS). Alternative is to use a docker volume to keep the db on the VM instead of shared with the host - but sharing with the host is very useful, so I hope this gets fixed soon.

  10. spoletumpr commented on May 12, 2026

    @spoletumpr

    Same problem here using a devcontainer.

  11. Sleepful commented on May 28, 2026

    @Sleepful
    Contributor

    Same issue running inside docker with

      -v "$HOME/.local/share/opencode:/root/.local/share/opencode" \
      -v "$HOME/.local/state/opencode:/root/.local/state/opencode" \
    

    which is really cool so it is a shame it fails : (

    Running opencode from source and using @zcutlip guideline was a quick fix.

  12. zcutlip commented on May 28, 2026

    @zcutlip

    I'm curious if everyone using this is on macOS as a host. Whether via Docker desktop, Colima, Apple Containers, or some other virtualization/containerization stack.

    I believe there may be a bug in the Apple Virtualization Framework that results in data inconsistency/corruption when guest and host have a shared folder.

    Most interesting is if anyone experiencing this is not using macOS at all.

  13. GaryGong27 commented on Jul 10, 2026

    @GaryGong27

    Seeing the same Project.fromDirectory / db.ts corruption path on 1.17.15–1.17.18 (macOS + Linux, local disk — not only NFS), with an additional trigger worth noting: cross-version schema drift on the shared opencode.db.

    Because the DB is shared by every opencode binary on the machine, if one version migrates the schema forward (drops/renames a column) and an older version then queries the old column at boot, you get:

    no such column: <col>   (e.g. project.name)
    

    → the server worker crash-loops and the interactive TUI renders a blank screen with no visible error (the error is only in ~/.local/share/opencode/log).

    Two asks:

    1. Surface a fatal, user-visible DB error (and exit non-zero) instead of a blank TUI.
    2. Guard schema migrations against a second live writer / older-version binary sharing the same DB+WAL.
  14. oldantique commented on Jul 11, 2026

    @oldantique

    Workaround that's been solid for me: give each process its own data dir via XDG_DATA_HOME

    I was hitting the locking/corruption intermittently when fanning out multiple one-shot opencode run calls in parallel. Root cause is the same one discussed here: every process on the machine shares the single global DB at ~/.local/share/opencode/opencode.db.

    Since the config (~/.config/opencode) and the models cache (~/.cache/opencode) resolve independently of the data dir, you can isolate just the SQLite side per process:

    d=$(mktemp -d)
    mkdir -p "$d/share/opencode"
    cp ~/.local/share/opencode/auth.json "$d/share/opencode/" 2>/dev/null || true  # only needed if you use `opencode auth login`
    XDG_DATA_HOME="$d/share" XDG_STATE_HOME="$d/state" opencode run "your prompt"
    rm -rf "$d"

    Wrapped, preserving the exit code:

    ocp() {
      local d rc
      d=$(mktemp -d)
      mkdir -p "$d/share/opencode"
      cp ~/.local/share/opencode/auth.json "$d/share/opencode/" 2>/dev/null || true
      XDG_DATA_HOME="$d/share" XDG_STATE_HOME="$d/state" opencode run "$@"
      rc=$?
      rm -rf "$d"
      return $rc
    }
    
    ocp "task A" > a.md &
    ocp "task B" > b.md &
    wait

    Why it works / what it costs:

    • each process gets a fresh, empty DB (instant to create), so cross-process locking and write-write corruption become impossible by construction;
    • config and the models cache stay shared — no re-fetch;
    • auth.json lives in the data dir, hence the cp above (re: @Miskamyasa's point). If your providers use env-var keys ({env:...} in opencode.json), you can drop that line;
    • as a bonus it sidesteps two other failure modes in this thread: SQLite never touches the NFS/virtio-shared home if $TMPDIR is a local disk, and the fresh per-run DB can't hit the cross-version schema-drift case @GaryGong27 described.

    Verified on v1.17.7 (Linux): 8 concurrent opencode run one-shots → 8/8 correct outputs, zero database is locked / locking protocol errors.

    Caveats:

    • only suitable for parallel one-shot runs — the session is written to the throwaway dir, so --continue / --session won't find it afterwards. Anything needing session continuity should keep the default dir (and run sequentially);
    • this is a workaround, not a fix — per-project DBs and/or PRAGMA busy_timeout as proposed above would be the proper solution.
  15. akostadinov commented on Aug 28, 2026

    @akostadinov

    Yes, please allow overriding OPENCODE_DATA_DIR as an environment variable as well as a configuration option. Overriding the whole XDG_DATA_HOME is super heavy handed.

  16. efirvida commented on Sep 21, 2026

    @efirvida

    Additional occurrence with WAL-generation forensics — NFSv4 home, killed process and concurrent instances, two incidents (Sep 19 & 20, 2026). Same defect class as this issue; adding evidence it currently lacks.

    Environment

    • Linux (RHEL 8.8, kernel 4.18) x86_64; home directory mounted from an enterprise NAS (Isilon) via NFSv4.0: vers=4.0,hard,rsize=1048576,wsize=1048576,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none.
    • Shared store at ~/.local/share/opencode/opencode.db (SQLite WAL mode, Drizzle/bun-sqlite stack).

    Triggers observed

    1. The opencode process was killed mid-write, then a new instance opened the same store.
    2. Concurrent opencode instances on the same NFS-mounted data dir.

    Corruption evidence (salvaged copy)

    • PRAGMA quick_check:
    *** in database main ***
    Tree 23 page 23: btreeInitPage() returns error code 11
    Tree 43 page 86 cell 25: Extends off end of page
    Tree 43 page 86 cell 24: Extends off end of page
    Tree 43 page 86 cell 23: Extends off end of page
    wrong # of entries in index sqlite_autoindex_project_1
    wrong # of entries in index sqlite_autoindex_event_sequence_1
    wrong # of entries in index event_aggregate_type_seq_idx
    wrong # of entries in index event_aggregate_seq_idx
    
    • Main DB file is structurally intact ("SQLite format 3" header, 49,464 pages), so corruption is concentrated in the WAL/SHM pair plus scattered btree pages — same pattern as the "database disk image is malformed" / "file is not a database" failures reported here.

    WAL forensics (quarantined -wal, 25,717,072 bytes, 6,242 frames @ 4096)

    • Header unparseable: page_size field reads 0x002de218 while frame geometry proves the real page size is 4096.
    • Header checksum does not validate under either endianness.
    • 8 distinct salt generations physically interleaved in one file (healthy WAL has exactly one salt pair): runs of 459/549/3/1/12/2/104/5112 frames with different salts — i.e. frames from multiple WAL generations written into the same file non-sequentially.
    • 0/200 sampled frames validate under any header-derived checksum seed → the WAL is unrecoverable by SQLite design; sessions/history/context written since the last checkpoint are lost.

    Reproducibility of the fingerprint
    The same interleaved-salt fingerprint (7 salts / 1,676 frames, unparseable header, 0/200 frames valid) is present in a -wal on a local ext4 machine after a kill — there the main DB stayed intact because SQLite ignored the broken WAL. So the pattern is not NFS-exclusive, but only the NFS deployment produced main-DB btree corruption and data loss.

    Impact

    • Sessions, history, and context lost; app reports file is not a database / blank UI on next start.
    • Manual salvage required: backup copy + sqlite3 .recover SQL dump (~189 MB) + rebuild; failure is silent until restart.

    Suggested hardening (additions to this thread)

    1. Detect a network filesystem for the data dir (statfs NFS magic 0x6969, or CIFS/SMB) and either refuse WAL (journal_mode=DELETE + synchronous=FULL) or fail loudly with an actionable message.
    2. App-level single-instance lock on the data dir so a second process cannot share one store over NFS (cross-host and cross-process).
    3. Startup integrity probe (PRAGMA quick_check) with automatic quarantine of -wal/-shm and recreation instead of crashing on file is not a database.
    4. PRAGMA busy_timeout + retry (ties into Concurrent opencode processes on one database fail prompts with "Failed to execute statement" (SQLITE_BUSY) #47566 / Daily "Failed to execute statement" on macOS with concurrent opencode processes (1.18.30, busy_timeout=0, 21 GB DB) #48416).

    Quarantined -wal/-shm and DB copies are retained and available for maintainer analysis.

  17. thdxr commented on Oct 11, 2026

    @thdxr
    Member

    Thanks for the report! OpenCode 2.0 is now out and development is focused on V2, so we're closing issues filed against OpenCode 1. Please upgrade (opencode upgrade --major, details: https://anoma.ly/notes/opencode-2-0), and if this is still relevant on 2.0, reopen this issue or open a new one with your 2.0 version.

    — from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions