Skip to content

logs_2.sqlite-wal grows without bound into tens of GB #28997

Open

Description

@th317erd

What version of Codex CLI is running?

codex-cli 0.140.0

codex doctor reports 0.141.0 is available, but this was observed on 0.140.0.

Also observed locally:

codex-cli 0.140.0
codex-cli 0.138.0  # codext wrapper

What platform is your computer?

Linux x86_64, Ubuntu 24.04 noble.

Runtime/install from codex doctor:

Codex Doctor v0.140.0 · linux-x86_64
runtime: npm
Node path: ~/.nvm/versions/node/v24.13.0/...
terminal: Alacritty
auth: ChatGPT auth

Which model were you using?

gpt-5.5

What issue are you seeing?

Codex SQLite log WAL files are growing without bound into many GB/tens of GB during normal local use. The main symptom is enormous logs_2.sqlite-wal files, while the base logs_2.sqlite database remains comparatively small.

This appears to be a default Codex log/state behavior, not something caused by a custom log setting. My setup uses multiple CODEX_HOME profile directories (~/.codex-work and ~/.codex-personal) in addition to the default ~/.codex, but that only relocates Codex state. The default config in ~/.codex/config.toml has not changed any log or SQLite settings. It only contains a trusted project entry. No log_dir or sqlite_home override is configured there.

The Codex docs say local state is stored under CODEX_HOME, defaulting to ~/.codex, and mention logs/caches as per-user state. They also document optional log_dir and sqlite_home settings. So this should affect a standard ~/.codex setup the same way; the profile dirs here just made the issue easier to isolate.

Concrete evidence

Earlier in this session, before cleanup:

~/.codex-work/logs_2.sqlite-wal      34,848,694,552 bytes
~/.codex-personal/logs_2.sqlite-wal  12,168,609,552 bytes
logs_2.sqlite-shm sidecars           about 20-25 MB
logs_2.sqlite base DB                about 115,650,560 bytes

After deleting/moving aside WAL/SHM sidecars and restarting Codex, the WALs reappeared and began growing again. A later check showed:

/home/<user>/.codex-personal/logs_2.sqlite      115,650,560 bytes
/home/<user>/.codex-personal/logs_2.sqlite-wal  1,693,608,432 bytes
/home/<user>/.codex-personal/logs_2.sqlite-shm  3,309,568 bytes

/home/<user>/.codex-work/logs_2.sqlite          115,650,560 bytes
/home/<user>/.codex-work/logs_2.sqlite-wal      19,184,359,792 bytes
/home/<user>/.codex-work/logs_2.sqlite-shm      37,257,216 bytes

/home/<user>/.codex/logs_2.sqlite               14,266,368 bytes
/home/<user>/.codex/logs_2.sqlite-wal           8,207,072 bytes
/home/<user>/.codex/logs_2.sqlite-shm           32,768 bytes

At the same time, profile directory sizes were dominated by these SQLite sidecars:

~/.codex-work      23G
~/.codex-personal  2.8G
~/.codex            104M

codex doctor reported the log DB itself as healthy:

state: databases healthy
log DB ~/.codex-personal/logs_2.sqlite (file) · integrity ok

But it did not warn about the enormous WAL file.

Impact

  • Normal backup/rsync of Codex state attempted to copy tens of GB of volatile SQLite WAL files.
  • The huge WAL files consumed significant disk space and made simple profile synchronization unexpectedly expensive.
  • Deleting the WAL while Codex processes are live does not actually reclaim disk until those processes exit because the file remains open.
  • Attempting to truncate open deleted WAL file descriptors caused active Codex instances to crash immediately.
  • The current workaround is to exclude *.sqlite-wal and *.sqlite-shm from rsync/backup and only clean/checkpoint these files when Codex is not running.

Steps to reproduce

  1. Use Codex CLI/app normally for an extended local coding session, especially with substantial tool output.

  2. Inspect the Codex state directory:

    find "$CODEX_HOME" -maxdepth 1 \( -name '*sqlite-wal' -o -name '*sqlite-shm' -o -name 'logs_2.sqlite' \) -printf '%p %s bytes\n' | sort
  3. Compare logs_2.sqlite-wal size with logs_2.sqlite.

Expected result: WAL remains bounded, periodically checkpointed, or is truncated on clean shutdown.

Actual result: logs_2.sqlite-wal can grow into tens of GB while the base log DB remains around ~100 MB.

Expected behavior

  • Codex should not allow logs_2.sqlite-wal to grow without a practical bound.
  • The logging/state layer should periodically checkpoint/truncate SQLite WAL files or configure WAL autocheckpointing appropriately.
  • Large tool output should be capped, summarized, or rotated before it can accumulate unboundedly in persistent local log state.
  • codex doctor should warn when logs_2.sqlite-wal or other Codex SQLite sidecars exceed a reasonable threshold.
  • Ideally, Codex should expose documented retention/size controls for local logs and SQLite-backed runtime state.

Related issues

This may be related to, but is more specific than:

This issue is specifically about unbounded logs_2.sqlite-wal growth under default-style Codex state behavior.

Activity

  1. github-actions commented on Jun 18, 2026

    @github-actions
    Contributor
  2. th317erd commented on Jun 18, 2026

    @th317erd
    Author

    Correction to the original evidence section: the peak pre-cleanup observation was roughly 65 GB of SQLite log WAL files across the two active Codex profile/session folders. The smaller per-file numbers originally listed there were from a later post-cleanup/regrowth check and should not have been presented as the pre-cleanup peak. I edited the issue body to separate the peak observation from the later regrowth evidence.

  3. th317erd commented on Jun 19, 2026

    @th317erd
    Author

    Additional observation: a closed/stale Codex work-profile log WAL just measured at 219G / 234,157,858,816 bytes: /home/<user>/.codex-work/logs_2.sqlite-wal. At that point the user's /home filesystem was full: 656G size, 623G used, 40M free, 100% used. The file was not held open by currently running Codex processes, so removing only the closed .codex-work/logs_2.sqlite-wal and .codex-work/logs_2.sqlite-shm restored free space to 219G available. Separately, currently running codext app-server/remotes were attached to .codex-personal/logs_2.sqlite-wal; three remote worker processes each showed roughly 170GB cumulative writes in /proc//io while the active personal WAL was about 1.6G at inspection time. This confirms the issue can become disk-filling, not just inconvenient for backups.

  4. th317erd commented on Jun 19, 2026

    @th317erd
    Author

    Follow-up DB-content scan from the surviving base log DBs found a likely self-amplifying source. The schema is a small log DB: logs(id, ts, ts_nanos, level, target, feedback_log_body, module_path, file, line, thread_id, process_uuid, estimated_bytes), with WAL journaling enabled. The base DBs are only about 111 MB each, but they contain many TRACE rows where Codex logs file watcher events for its own log DB/WAL changes.\n\nExamples from .codex-work/logs_2.sqlite:\n\n- total logs rows: 42,534\n- TRACE rows: 38,537\n- rows with inotify event: 35,571\n- rows mentioning logs_2.sqlite-wal: 26,401\n- repeated recent sample: inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some(\"logs_2.sqlite-wal\") }\n\nExamples from .codex-personal/logs_2.sqlite:\n\n- total logs rows: 48,872\n- TRACE rows: 44,153\n- rows with inotify event: 38,580\n- rows mentioning logs_2.sqlite-wal: 25,397\n- recent samples include repeated inotify event ... name: Some(\"logs_2.sqlite\") rows.\n\nThis looks like a feedback loop: Codex logs that its own log DB/WAL changed; that insert changes the WAL again; the watcher observes another modify event; then another TRACE row is written. Because the DB is in WAL mode and checkpoint/truncation is not keeping the WAL bounded, a small 111 MB base DB can generate a 219 GB logs_2.sqlite-wal. There are also large TRACE payload rows from websocket/frame logging, but the inotify self-logging loop is the clearest disk-filling mechanism found so far.

  5. th317erd commented on Jun 19, 2026

    @th317erd
    Author

    Additional sanitized findings from a deeper investigation:

    The strongest evidence now points to a self-watch / self-log loop around $CODEX_HOME.

    Likely loop

    1. Codex writes a diagnostic row to logs_2.sqlite.
    2. SQLite updates logs_2.sqlite-wal.
    3. A Codex inotify watcher sees logs_2.sqlite-wal modified.
    4. Codex records a TRACE row for that inotify event into the same logs_2.sqlite.
    5. That write modifies logs_2.sqlite-wal again.
    6. Repeat.

    Important environment note

    The newest evidence was collected while the affected session was being run through the codext fork, which vendors/runs a codex binary. That means this may be fork-specific, fork-amplified, or caused by an interaction between upstream Codex code and codext's app-server/remote process model. I am posting this here because the DB/schema/logging/watch behavior is Codex-derived, but I am also filing the same sanitized report against Loongphy/codext.

    Observed state

    Sanitized paths:

    CODEX_HOME=/home/<user>/.codex-work
    SQLite DB=/home/<user>/.codex-work/logs_2.sqlite
    SQLite WAL=/home/<user>/.codex-work/logs_2.sqlite-wal
    

    The current post-cleanup WAL still grew back to roughly 12 GB:

    11,884,873,392  /home/<user>/.codex-work/logs_2.sqlite-wal
    115,650,560     /home/<user>/.codex-work/logs_2.sqlite
    23,101,440      /home/<user>/.codex-work/logs_2.sqlite-shm
    

    Earlier growth samples during active sessions showed severe WAL growth:

    2,492,175,672 -> 2,842,709,392 bytes in 10 seconds
    5,953,070,432 -> 6,316,977,672 bytes in 10 seconds
    8,921,312,072 -> 9,101,376,672 bytes in 5 seconds
    

    This is after a previous stale/closed WAL reached 219 GB and filled the /home filesystem.

    SQLite log-table evidence

    Counts from the affected logs_2.sqlite:

    total_rows=47,565
    TRACE=42,794
    INFO=3,533
    DEBUG=864
    WARN=356
    ERROR=18
    inotify_rows=39,532
    logs_2.sqlite-wal mentions=28,727
    

    Top repeated inotify messages:

    28,699  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite-wal") }
     8,049  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("logs_2.sqlite") }
       147  inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some("state_5.sqlite-wal") }
    

    There are also unrelated file-open watcher events for system files such as ld.so.cache, locale.alias, and passwd, which appear to come from a separate /etc watch. The disk-filling loop is the one involving logs_2.sqlite*.

    Kernel inotify evidence

    The active affected process had an inotify file descriptor with:

    FD 30 anon_inode:inotify
      inotify wd:1 ino:c6860b ...
    

    The inode maps to $CODEX_HOME:

    hex c6860b == decimal 13010443
    13010443 /home/<user>/.codex-work
    

    The repeated SQLite log rows also use WatchDescriptor { id: 1, ... }, and the row names are logs_2.sqlite-wal / logs_2.sqlite. That ties the logged file events directly to a watcher rooted at $CODEX_HOME, not just to a random project directory.

    Open file holders

    The affected codext/vendored codex process held open handles to:

    /home/<user>/.codex-work/logs_2.sqlite
    /home/<user>/.codex-work/logs_2.sqlite-wal
    /home/<user>/.codex-work/logs_2.sqlite-shm
    

    Other active codext app-server/remote processes held handles to a separate profile's logs_2.sqlite-wal.

    Process model clue

    The process tree included:

    node .../bin/codext ... app-server --listen ws://127.0.0.1:<port>
    .../codex ... app-server --listen ws://127.0.0.1:<port>
    node .../bin/codext ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
    .../codex ... --remote ws://127.0.0.1:<port> -C /home/<user>/Projects/<repo>
    node .../bin/codext ... --sandbox danger-full-access --ask-for-approval never
    .../codex ... --sandbox danger-full-access --ask-for-approval never
    

    The exact project names have been omitted intentionally. Non-Codex dev-server processes running in one project did not hold handles to logs_2.sqlite*; only Codex/codext processes did.

    Trust/root clue

    The current Git repo root resolved correctly to a project directory under /home/<user>/Projects/<repo>, but the UI reportedly displayed a trust warning for /home/<user>, not the project root. If a broad home directory becomes a trusted/watch surface, it can include $CODEX_HOME and therefore Codex's own SQLite state.

    Even without that clue, the kernel inotify evidence above shows a watcher rooted at /home/<user>/.codex-work.

    Relevant source areas

    These upstream files look relevant:

    codex-rs/file-watcher/src/lib.rs
    codex-rs/app-server/src/fs_watch.rs
    codex-rs/app-server/src/skills_watcher.rs
    codex-rs/tui/src/onboarding/onboarding_screen.rs
    

    Specifically:

    • file-watcher can watch requested paths and can fall back to the nearest existing ancestor.
    • app-server exposes fs/watch.
    • skills roots can be watched recursively.
    • trust onboarding falls back to cwd if no Git root is resolved.

    Expected behavior

    Codex should not log file watcher events for its own SQLite diagnostic/state files into the same SQLite log sink.

    At minimum, file watcher trace logging should suppress:

    logs_2.sqlite
    logs_2.sqlite-wal
    logs_2.sqlite-shm
    state_5.sqlite
    state_5.sqlite-wal
    state_5.sqlite-shm
    goals_1.sqlite*
    memories_1.sqlite*
    

    Better fixes:

    1. Never watch $CODEX_HOME as a filesystem watch root unless explicitly required.
    2. Never log watcher events for Codex's own SQLite files into the SQLite log sink.
    3. Respect log-level filtering before inserting TRACE rows into logs_2.sqlite.
    4. Add WAL size limits, checkpointing, rotation, or emergency safeguards so diagnostic logs cannot consume hundreds of GB.
    5. If a broad home directory trust target is selected, exclude Codex state directories from any watch surfaces.

    Local mitigation being tested

    The local mitigation is to move SQLite-backed runtime state outside the watched CODEX_HOME path:

    sqlite_home = "/home/<user>/.local/share/codex-sqlite/work"

    This was added after the issue was observed. Already-running Codex/codext processes can continue holding the old logs_2.sqlite* files until restarted.

  6. th317erd commented on Jun 19, 2026

    @th317erd
    Author

    Cross-filed the sanitized fork-specific report against Loongphy/codext because the latest evidence was collected from codext/vendored-codex processes: Loongphy#11

  7. insilications commented on Jun 19, 2026

    @insilications

    This kind of program behavior is disrespectful in general, especially toward users with SSDs.

  8. th317erd commented on Jun 19, 2026

    @th317erd
    Author

    Follow-up after comparing the official OpenAI repo against the Codext fork and testing a fix:

    I now think the concrete trigger I hit is not directly an OpenAI/Codex bug. The direct self-watch loop appears fork-specific in Loongphy/codext: that fork adds an AuthWatch that watches the top-level CODEX_HOME directory for auth.json changes. Since the SQLite log DB normally also lives under CODEX_HOME, WAL writes can be observed by that watcher, and raw notify TRACE logs can be written back into the same SQLite log DB.

    Fork-specific issue:

    Loongphy#11

    Fix PR:

    Loongphy#12

    That said, I think the underlying risk is still present in OpenAI/Codex's implementation: the SQLite log sink persists TRACE logs by default, and the shared log_db.rs did not filter raw notify/inotify compatibility logs. I did not find an official OpenAI/Codex production path that automatically watches CODEX_HOME itself, so the immediate trigger seems absent upstream. But if any current or future official watcher targets the SQLite directory, or a client asks app-server fs/watch to watch that location, the same class of feedback loop could become user-visible.

    So this can probably be closed as "not directly reproduced in upstream OpenAI/Codex" if maintainers agree, but I would still recommend hardening the SQLite log sink so it does not persist raw file-watcher backend TRACE/DEBUG noise in a very damaging leaking fs-notify loop.

  9. denispol commented on Jun 25, 2026

    @denispol

    Fix bounds the WAL: journal_size_limit=16MiB + wal_autocheckpoint=400, plus a periodic best-effort TRUNCATE checkpoint (previously only a startup PASSIVE checkpoint): https://github.com/denispol/codex/pull/1

  10. cosgroveb commented on Jul 3, 2026

    @cosgroveb

    I hit these but my sqlite db size was genuinely nutty 😆

    % du -h --apparent-size ~/codex-db-backup/logs_2.sqlite*
    1.1G    /home/bcosgrove/codex-db-backup/logs_2.sqlite
    1.3G    /home/bcosgrove/codex-db-backup/logs_2.sqlite-shm
    1.1T    /home/bcosgrove/codex-db-backup/logs_2.sqlite-wal
    

    10s of GB? Pshaw.

  11. bbingz commented on Jul 24, 2026

    @bbingz

    Analysis (community)

    Thanks for tracking this — attempting a residual-focused analysis aligned with docs/contributing.md (invitation-only: analysis in-thread, no unsolicited PR).

    • Issue under review: logs_2.sqlite-wal grows without bound into tens of GB #28997 — logs_2.sqlite-wal grows without bound into tens of GB
    • Theme / radar cluster: logs_2.sqlite-wal multi-GB growth (logs_2.sqlite SSD)
    • Reporter posture: static + local evidence where available; not claiming a maintainer invite.

    Observed (from issue + public residual signal)

    What version of Codex CLI is running? codex-cli 0.140.0 codex doctor reports 0.141.0 is available, but this was observed on 0.140.0. Also observed locally: text codex-cli 0.140.0 codex-cli 0.138.0 # codext wrapper ### What platform is your computer? Linux x86_64, Ubuntu 24.04 noble. Runtime/install from codex doctor: text Codex Doctor v0.140.0 · linux-x86_64 runtime: npm Node path: ~/.nvm/versions/node/v24.13.0/... terminal: Alacritty auth: ChatGPT auth ### Which mode

    Local residual measurement (2026-07-24, this reporter)

    On a host running codex-cli 0.146.0-alpha.6 (newer than 0.142–0.145 residual reports):

    Signal-14 INTERIM evidence (from qwen_s14 mid-flight)

    captured: 2026-07-24T15:10:05Z

    • /Users/bing/.codex/logs_2.sqlite size=1778548736 mtime=Fri Jul 24 23:09:38 2026
    • /Users/bing/.codex/logs_2.sqlite-wal size=30405632 mtime=Fri Jul 24 23:10:04 2026
    • codex version: codex-cli 0.146.0-alpha.6

    level counts (ro)

    TRACE|94794
    DEBUG|21528
    INFO|19618
    WARN|796
    ERROR|370
    

    rows/max_ts: 137106|2026-07-24 23:10:05

    This is not a claim that every install still burns TB/year — it is a version-pinned observation that TRACE still lands in the SQLite sink after the #28224-era mitigations on a current alpha, consistent with residual issues still open.

    Root-cause hypothesis (to be confirmed on main)

    1. Feedback/log pipeline still admits TRACE (or default level too verbose) into logs_2.sqlite.
    2. WAL/checkpoint policy does not cap growth under idle or streaming load.
    3. Partial cut of TRACE paths may leave high-volume targets writing.

    High-level fix outline (not a PR)

    • Default sink level: never store TRACE in SQLite unless explicitly enabled.
    • Honor RUST_LOG / app log level end-to-end into the SQLite writer.
    • Hard caps: max DB size / WAL size / rotation with user-visible warning.
    • Fail-first tests: write-rate budget under idle + streaming fixtures.

    Tests

    • Unit: writer rejects TRACE when configured level is INFO.
    • Integration: after N seconds idle with default config, row growth rate < bound.

    Questions for maintainers

    1. Is the intended long-term policy documented for this area (especially sandbox gitdir / log TRACE defaults)?
    2. Preferred residual anchor issue if several open threads describe the same mechanism?
    3. If the high-level outline above is roughly aligned, would an invited PR with failing-first tests be welcome later — or is an internal fix preferred?

    Happy to refine with more measurements or code pointers. No unsolicited PR from me.

  12. bbingz commented on Jul 24, 2026

    @bbingz

    Analysis (community)

    Residual analysis for #28997 (logs_2.sqlite-wal unbounded growth / disk-fill) under docs/contributing.md — invitation-only, no unsolicited PR.

    This is intentionally not a restatement of the TRACE-default residual already filed on #17320. Focus here is the sink budget / WAL lifecycle gap that lets a multi-GB WAL appear even when the base DB is smaller.

    Deterministic evidence (code + runtime)

    Code on current main (5dd992acd3):

    • SQLite log file constant: codex-rs/state/src/sqlite.rs LOGS_DB_FILENAME = "logs_2.sqlite".
    • Async sink knobs in codex-rs/state/src/log_db.rs:
      • LOG_QUEUE_CAPACITY = 512
      • LOG_BATCH_SIZE = 128
      • LOG_FLUSH_INTERVAL = Duration::from_secs(2) (periodic commit pressure)
    • LogDbLayer::on_event (≈L204) builds a LogEntry and try_sends into the inserter (≈L151–L152, L249). There is no max-bytes / rotate / hard WAL checkpoint budget adjacent to the flush path.
    • default_filter() still defaults to TRACE (L55) — that is the volume source; logs_2.sqlite-wal grows without bound into tens of GB #28997’s residual is the missing ceiling once volume exists.

    Runtime on this host (2026-07-24, codex-cli 0.146.0-alpha.6):

    • ~/.codex/logs_2.sqlite size 1778548736 bytes (~1.66 GiB)
    • ~/.codex/logs_2.sqlite-wal size 30405632 bytes (~29 MiB still dirty under load)
    • Level histogram (sqlite3 RO): TRACE|94907 DEBUG|21547 INFO|19528 WARN|796 ERROR|372 (total rows ≈137150)
    • Community on logs_2.sqlite-wal grows without bound into tens of GB #28997 reported tens–hundreds of GB WAL and a self-watch feedback loop; my local numbers show the sink is still actively dirty on 0.146.x even without that extreme.

    Root-cause hypothesis

    Two cooperating defects:

    1. High event fan-in (TRACE-default + streaming targets) continuously dirties the WAL via 2s batch flushes.
    2. No enforced max DB/WAL bytes, rotation, or aggressive checkpoint-on-threshold → pathological growth can fill disks (as reported).

    This is a resource-governance residual, not “just lower the log level”.

    High-level fix outline (not a PR)

    1. Hard-cap logs_2.sqlite + WAL (truncate/rotate + user-visible warning when threshold hit).
    2. Checkpoint/truncate WAL when size exceeds N MB; never rely on process exit alone.
    3. Optionally suppress file-watcher TRACE for $CODEX_HOME/logs_2.sqlite* to break self-amplifying loops reported on this issue.
    4. Fail-first integration: generate load, assert WAL stays under bound; regression if WAL exceeds cap after T minutes.

    Questions for maintainers

    1. Is there an intentional internal budget for logs_2.sqlite / WAL size today?
    2. Should rotation live in the state crate next to LogDbLayer, or in app-server packaging?
    3. If the outline aligns, would an invited PR with failing-first size-cap tests be welcome?

    Happy to refine measurements. No unsolicited PR from me.

  13. mingisrookie commented on Jul 26, 2026

    @mingisrookie

    Related data point — even after log volume is addressed, the space is never reclaimed:

    On Codex Desktop 26.721.4979.0 / app-server 0.146.0-alpha.3.1 (native Windows), ~/.codex/logs_2.sqlite was 650 MB with 0 rows in the logs table:

    PRAGMA page_count → 166,493 pages
    PRAGMA freelist_count → 166,280 pages (99.9% empty)
    SELECT COUNT(*) FROM logs → 0
    

    So the retention/cleanup did run at some point and deleted the rows, but the file is never VACUUMed, leaving a 650 MB empty shell that still gets opened/checkpointed on every launch.

    Additionally, an abandoned 257 MB logs_2.sqlite (35,938 rows) was left behind at the pre-migration location ~/.codex/sqlite/logs_2.sqlite, untouched since the DB moved to ~/.codex/ — the migration didn't remove the old file.

    Suggestion: run VACUUM (or PRAGMA incremental_vacuum) after retention deletes, and clean up the old-location DB after migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIbugSomething isn't workingperformance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions