Repository navigation
logs_2.sqlite-wal grows without bound into tens of GB #28997
Description
What version of Codex CLI is running?
codex-cli 0.140.0
codex doctor reports 0.141.0 is available, but this was observed on 0.140.0.
Also observed locally:
codex-cli 0.140.0
codex-cli 0.138.0 # codext wrapper
What platform is your computer?
Linux x86_64, Ubuntu 24.04 noble.
Runtime/install from codex doctor:
Codex Doctor v0.140.0 · linux-x86_64
runtime: npm
Node path: ~/.nvm/versions/node/v24.13.0/...
terminal: Alacritty
auth: ChatGPT auth
Which model were you using?
gpt-5.5
What issue are you seeing?
Codex SQLite log WAL files are growing without bound into many GB/tens of GB during normal local use. The main symptom is enormous logs_2.sqlite-wal files, while the base logs_2.sqlite database remains comparatively small.
This appears to be a default Codex log/state behavior, not something caused by a custom log setting. My setup uses multiple CODEX_HOME profile directories (~/.codex-work and ~/.codex-personal) in addition to the default ~/.codex, but that only relocates Codex state. The default config in ~/.codex/config.toml has not changed any log or SQLite settings. It only contains a trusted project entry. No log_dir or sqlite_home override is configured there.
The Codex docs say local state is stored under CODEX_HOME, defaulting to ~/.codex, and mention logs/caches as per-user state. They also document optional log_dir and sqlite_home settings. So this should affect a standard ~/.codex setup the same way; the profile dirs here just made the issue easier to isolate.
Concrete evidence
Earlier in this session, before cleanup:
~/.codex-work/logs_2.sqlite-wal 34,848,694,552 bytes
~/.codex-personal/logs_2.sqlite-wal 12,168,609,552 bytes
logs_2.sqlite-shm sidecars about 20-25 MB
logs_2.sqlite base DB about 115,650,560 bytes
After deleting/moving aside WAL/SHM sidecars and restarting Codex, the WALs reappeared and began growing again. A later check showed:
/home/<user>/.codex-personal/logs_2.sqlite 115,650,560 bytes
/home/<user>/.codex-personal/logs_2.sqlite-wal 1,693,608,432 bytes
/home/<user>/.codex-personal/logs_2.sqlite-shm 3,309,568 bytes
/home/<user>/.codex-work/logs_2.sqlite 115,650,560 bytes
/home/<user>/.codex-work/logs_2.sqlite-wal 19,184,359,792 bytes
/home/<user>/.codex-work/logs_2.sqlite-shm 37,257,216 bytes
/home/<user>/.codex/logs_2.sqlite 14,266,368 bytes
/home/<user>/.codex/logs_2.sqlite-wal 8,207,072 bytes
/home/<user>/.codex/logs_2.sqlite-shm 32,768 bytes
At the same time, profile directory sizes were dominated by these SQLite sidecars:
~/.codex-work 23G
~/.codex-personal 2.8G
~/.codex 104M
codex doctor reported the log DB itself as healthy:
state: databases healthy
log DB ~/.codex-personal/logs_2.sqlite (file) · integrity ok
But it did not warn about the enormous WAL file.
Impact
- Normal backup/rsync of Codex state attempted to copy tens of GB of volatile SQLite WAL files.
- The huge WAL files consumed significant disk space and made simple profile synchronization unexpectedly expensive.
- Deleting the WAL while Codex processes are live does not actually reclaim disk until those processes exit because the file remains open.
- Attempting to truncate open deleted WAL file descriptors caused active Codex instances to crash immediately.
- The current workaround is to exclude
*.sqlite-waland*.sqlite-shmfrom rsync/backup and only clean/checkpoint these files when Codex is not running.
Steps to reproduce
-
Use Codex CLI/app normally for an extended local coding session, especially with substantial tool output.
-
Inspect the Codex state directory:
find "$CODEX_HOME" -maxdepth 1 \( -name '*sqlite-wal' -o -name '*sqlite-shm' -o -name 'logs_2.sqlite' \) -printf '%p %s bytes\n' | sort
-
Compare
logs_2.sqlite-walsize withlogs_2.sqlite.
Expected result: WAL remains bounded, periodically checkpointed, or is truncated on clean shutdown.
Actual result: logs_2.sqlite-wal can grow into tens of GB while the base log DB remains around ~100 MB.
Expected behavior
- Codex should not allow
logs_2.sqlite-walto grow without a practical bound. - The logging/state layer should periodically checkpoint/truncate SQLite WAL files or configure WAL autocheckpointing appropriately.
- Large tool output should be capped, summarized, or rotated before it can accumulate unboundedly in persistent local log state.
codex doctorshould warn whenlogs_2.sqlite-walor other Codex SQLite sidecars exceed a reasonable threshold.- Ideally, Codex should expose documented retention/size controls for local logs and SQLite-backed runtime state.
Related issues
This may be related to, but is more specific than:
- Desktop launch can fail when logs_2.sqlite grows large: app-server SQLite pool times out during startup #27741, where a large
logs_2.sqlitecauses Desktop startup failures. - Heavy I/O activity from idle
codexprocesses. #20563, which reports heavy I/O from idle Codex processes.
This issue is specifically about unbounded logs_2.sqlite-wal growth under default-style Codex state behavior.
Activity
github-actions commentedon Jun 18, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
th317erd commentedon Jun 18, 2026
Correction to the original evidence section: the peak pre-cleanup observation was roughly 65 GB of SQLite log WAL files across the two active Codex profile/session folders. The smaller per-file numbers originally listed there were from a later post-cleanup/regrowth check and should not have been presented as the pre-cleanup peak. I edited the issue body to separate the peak observation from the later regrowth evidence.
th317erd commentedon Jun 19, 2026
Additional observation: a closed/stale Codex work-profile log WAL just measured at 219G / 234,157,858,816 bytes:
/home/<user>/.codex-work/logs_2.sqlite-wal. At that point the user's /home filesystem was full: 656G size, 623G used, 40M free, 100% used. The file was not held open by currently running Codex processes, so removing only the closed.codex-work/logs_2.sqlite-waland.codex-work/logs_2.sqlite-shmrestored free space to 219G available. Separately, currently running codext app-server/remotes were attached to.codex-personal/logs_2.sqlite-wal; three remote worker processes each showed roughly 170GB cumulative writes in /proc//io while the active personal WAL was about 1.6G at inspection time. This confirms the issue can become disk-filling, not just inconvenient for backups.th317erd commentedon Jun 19, 2026
Follow-up DB-content scan from the surviving base log DBs found a likely self-amplifying source. The schema is a small log DB:
logs(id, ts, ts_nanos, level, target, feedback_log_body, module_path, file, line, thread_id, process_uuid, estimated_bytes), with WAL journaling enabled. The base DBs are only about 111 MB each, but they contain many TRACE rows where Codex logs file watcher events for its own log DB/WAL changes.\n\nExamples from.codex-work/logs_2.sqlite:\n\n- total logs rows: 42,534\n- TRACE rows: 38,537\n- rows withinotify event: 35,571\n- rows mentioninglogs_2.sqlite-wal: 26,401\n- repeated recent sample:inotify event: Event { wd: WatchDescriptor { id: 1, fd: (Weak) }, mask: EventMask(MODIFY), cookie: 0, name: Some(\"logs_2.sqlite-wal\") }\n\nExamples from.codex-personal/logs_2.sqlite:\n\n- total logs rows: 48,872\n- TRACE rows: 44,153\n- rows withinotify event: 38,580\n- rows mentioninglogs_2.sqlite-wal: 25,397\n- recent samples include repeatedinotify event ... name: Some(\"logs_2.sqlite\")rows.\n\nThis looks like a feedback loop: Codex logs that its own log DB/WAL changed; that insert changes the WAL again; the watcher observes another modify event; then another TRACE row is written. Because the DB is in WAL mode and checkpoint/truncation is not keeping the WAL bounded, a small 111 MB base DB can generate a 219 GBlogs_2.sqlite-wal. There are also large TRACE payload rows from websocket/frame logging, but the inotify self-logging loop is the clearest disk-filling mechanism found so far.th317erd commentedon Jun 19, 2026
Additional sanitized findings from a deeper investigation:
The strongest evidence now points to a self-watch / self-log loop around
$CODEX_HOME.Likely loop
logs_2.sqlite.logs_2.sqlite-wal.logs_2.sqlite-walmodified.TRACErow for that inotify event into the samelogs_2.sqlite.logs_2.sqlite-walagain.Important environment note
The newest evidence was collected while the affected session was being run through the
codextfork, which vendors/runs acodexbinary. That means this may be fork-specific, fork-amplified, or caused by an interaction between upstream Codex code andcodext's app-server/remote process model. I am posting this here because the DB/schema/logging/watch behavior is Codex-derived, but I am also filing the same sanitized report againstLoongphy/codext.Observed state
Sanitized paths:
The current post-cleanup WAL still grew back to roughly 12 GB:
Earlier growth samples during active sessions showed severe WAL growth:
This is after a previous stale/closed WAL reached 219 GB and filled the
/homefilesystem.SQLite log-table evidence
Counts from the affected
logs_2.sqlite:Top repeated inotify messages:
There are also unrelated file-open watcher events for system files such as
ld.so.cache,locale.alias, andpasswd, which appear to come from a separate/etcwatch. The disk-filling loop is the one involvinglogs_2.sqlite*.Kernel inotify evidence
The active affected process had an inotify file descriptor with:
The inode maps to
$CODEX_HOME:The repeated SQLite log rows also use
WatchDescriptor { id: 1, ... }, and the row names arelogs_2.sqlite-wal/logs_2.sqlite. That ties the logged file events directly to a watcher rooted at$CODEX_HOME, not just to a random project directory.Open file holders
The affected
codext/vendoredcodexprocess held open handles to:Other active
codextapp-server/remote processes held handles to a separate profile'slogs_2.sqlite-wal.Process model clue
The process tree included:
The exact project names have been omitted intentionally. Non-Codex dev-server processes running in one project did not hold handles to
logs_2.sqlite*; only Codex/codext processes did.Trust/root clue
The current Git repo root resolved correctly to a project directory under
/home/<user>/Projects/<repo>, but the UI reportedly displayed a trust warning for/home/<user>, not the project root. If a broad home directory becomes a trusted/watch surface, it can include$CODEX_HOMEand therefore Codex's own SQLite state.Even without that clue, the kernel inotify evidence above shows a watcher rooted at
/home/<user>/.codex-work.Relevant source areas
These upstream files look relevant:
Specifically:
file-watchercan watch requested paths and can fall back to the nearest existing ancestor.fs/watch.cwdif no Git root is resolved.Expected behavior
Codex should not log file watcher events for its own SQLite diagnostic/state files into the same SQLite log sink.
At minimum, file watcher trace logging should suppress:
Better fixes:
$CODEX_HOMEas a filesystem watch root unless explicitly required.logs_2.sqlite.Local mitigation being tested
The local mitigation is to move SQLite-backed runtime state outside the watched
CODEX_HOMEpath:This was added after the issue was observed. Already-running Codex/codext processes can continue holding the old
logs_2.sqlite*files until restarted.th317erd commentedon Jun 19, 2026
Cross-filed the sanitized fork-specific report against Loongphy/codext because the latest evidence was collected from codext/vendored-codex processes: Loongphy#11
insilications commentedon Jun 19, 2026
This kind of program behavior is disrespectful in general, especially toward users with SSDs.
th317erd commentedon Jun 19, 2026
Follow-up after comparing the official OpenAI repo against the Codext fork and testing a fix:
I now think the concrete trigger I hit is not directly an OpenAI/Codex bug. The direct self-watch loop appears fork-specific in
Loongphy/codext: that fork adds anAuthWatchthat watches the top-levelCODEX_HOMEdirectory forauth.jsonchanges. Since the SQLite log DB normally also lives underCODEX_HOME, WAL writes can be observed by that watcher, and rawnotifyTRACE logs can be written back into the same SQLite log DB.Fork-specific issue:
Loongphy#11
Fix PR:
Loongphy#12
That said, I think the underlying risk is still present in OpenAI/Codex's implementation: the SQLite log sink persists TRACE logs by default, and the shared
log_db.rsdid not filter rawnotify/inotify compatibility logs. I did not find an official OpenAI/Codex production path that automatically watchesCODEX_HOMEitself, so the immediate trigger seems absent upstream. But if any current or future official watcher targets the SQLite directory, or a client asks app-serverfs/watchto watch that location, the same class of feedback loop could become user-visible.So this can probably be closed as "not directly reproduced in upstream OpenAI/Codex" if maintainers agree, but I would still recommend hardening the SQLite log sink so it does not persist raw file-watcher backend TRACE/DEBUG noise in a very damaging leaking fs-notify loop.
0xdevalias commentedon Jun 23, 2026
Potentially a duplicate of / related to:
Which should have been fixed by:
Which were released in:
But there still seem to be issues after that:
denispol commentedon Jun 25, 2026
Fix bounds the WAL:
journal_size_limit=16MiB+wal_autocheckpoint=400, plus a periodic best-effortTRUNCATEcheckpoint (previously only a startup PASSIVE checkpoint): https://github.com/denispol/codex/pull/1cosgroveb commentedon Jul 3, 2026
I hit these but my sqlite db size was genuinely nutty 😆
10s of GB? Pshaw.
bbingz commentedon Jul 24, 2026
Analysis (community)
Thanks for tracking this — attempting a residual-focused analysis aligned with docs/contributing.md (invitation-only: analysis in-thread, no unsolicited PR).
logs_2.sqlite SSD)Observed (from issue + public residual signal)
What version of Codex CLI is running?
codex-cli 0.140.0codex doctorreports0.141.0is available, but this was observed on0.140.0. Also observed locally:text codex-cli 0.140.0 codex-cli 0.138.0 # codext wrapper### What platform is your computer? Linux x86_64, Ubuntu 24.04 noble. Runtime/install fromcodex doctor:text Codex Doctor v0.140.0 · linux-x86_64 runtime: npm Node path: ~/.nvm/versions/node/v24.13.0/... terminal: Alacritty auth: ChatGPT auth### Which modeLocal residual measurement (2026-07-24, this reporter)
On a host running codex-cli 0.146.0-alpha.6 (newer than 0.142–0.145 residual reports):
Signal-14 INTERIM evidence (from qwen_s14 mid-flight)
captured: 2026-07-24T15:10:05Z
/Users/bing/.codex/logs_2.sqlitesize=1778548736 mtime=Fri Jul 24 23:09:38 2026/Users/bing/.codex/logs_2.sqlite-walsize=30405632 mtime=Fri Jul 24 23:10:04 2026codex-cli 0.146.0-alpha.6level counts (ro)
rows/max_ts: 137106|2026-07-24 23:10:05
This is not a claim that every install still burns TB/year — it is a version-pinned observation that TRACE still lands in the SQLite sink after the #28224-era mitigations on a current alpha, consistent with residual issues still open.
Root-cause hypothesis (to be confirmed on
main)logs_2.sqlite.High-level fix outline (not a PR)
RUST_LOG/ app log level end-to-end into the SQLite writer.Tests
Questions for maintainers
Happy to refine with more measurements or code pointers. No unsolicited PR from me.
bbingz commentedon Jul 24, 2026
Analysis (community)
Residual analysis for #28997 (
logs_2.sqlite-walunbounded growth / disk-fill) under docs/contributing.md — invitation-only, no unsolicited PR.This is intentionally not a restatement of the TRACE-default residual already filed on #17320. Focus here is the sink budget / WAL lifecycle gap that lets a multi-GB WAL appear even when the base DB is smaller.
Deterministic evidence (code + runtime)
Code on current main (
5dd992acd3):codex-rs/state/src/sqlite.rsLOGS_DB_FILENAME = "logs_2.sqlite".codex-rs/state/src/log_db.rs:LOG_QUEUE_CAPACITY = 512LOG_BATCH_SIZE = 128LOG_FLUSH_INTERVAL = Duration::from_secs(2)(periodic commit pressure)LogDbLayer::on_event(≈L204) builds aLogEntryandtry_sends into the inserter (≈L151–L152, L249). There is no max-bytes / rotate / hard WAL checkpoint budget adjacent to the flush path.default_filter()still defaults to TRACE (L55) — that is the volume source; logs_2.sqlite-wal grows without bound into tens of GB #28997’s residual is the missing ceiling once volume exists.Runtime on this host (2026-07-24,
codex-cli 0.146.0-alpha.6):~/.codex/logs_2.sqlitesize 1778548736 bytes (~1.66 GiB)~/.codex/logs_2.sqlite-walsize 30405632 bytes (~29 MiB still dirty under load)Root-cause hypothesis
Two cooperating defects:
This is a resource-governance residual, not “just lower the log level”.
High-level fix outline (not a PR)
logs_2.sqlite+ WAL (truncate/rotate + user-visible warning when threshold hit).$CODEX_HOME/logs_2.sqlite*to break self-amplifying loops reported on this issue.Questions for maintainers
logs_2.sqlite/ WAL size today?statecrate next toLogDbLayer, or in app-server packaging?Happy to refine measurements. No unsolicited PR from me.
mingisrookie commentedon Jul 26, 2026
Related data point — even after log volume is addressed, the space is never reclaimed:
On Codex Desktop
26.721.4979.0/ app-server0.146.0-alpha.3.1(native Windows),~/.codex/logs_2.sqlitewas 650 MB with 0 rows in thelogstable:So the retention/cleanup did run at some point and deleted the rows, but the file is never
VACUUMed, leaving a 650 MB empty shell that still gets opened/checkpointed on every launch.Additionally, an abandoned 257 MB
logs_2.sqlite(35,938 rows) was left behind at the pre-migration location~/.codex/sqlite/logs_2.sqlite, untouched since the DB moved to~/.codex/— the migration didn't remove the old file.Suggestion: run
VACUUM(orPRAGMA incremental_vacuum) after retention deletes, and clean up the old-location DB after migration.