Repository navigation
Codex SQLite feedback logs can write ~640 TB/year and rapidly consume SSD endurance #28224
Description
Update at Jun 23, 2026: the following 3 PRs are merged, it could avoid 85% logs(feedback from my codex), so let me close this issue.
Thanks @jif-oai for the fix.
- Stop logging every Responses WebSocket event #29432 (released in 0.142.0)
- Filter noisy targets from persistent logs #29457 (released in 0.142.0)
- Stop persisting bridged log events #29599 (will be released in 0.143.0)
Simple Workaround from @beskay #28224 (comment)
- Quit codex
- Run this command:
sqlite3 ~/.codex/logs_2.sqlite "CREATE TRIGGER IF NOT EXISTS
block_log_inserts BEFORE INSERT ON logs BEGIN SELECT RAISE(IGNORE);
END;"
Following is the original issue
Issue
Codex is continuously writing a large amount of data to the local SQLite feedback log database:
~/.codex/logs_2.sqlite~/.codex/logs_2.sqlite-wal~/.codex/logs_2.sqlite-shm
On my machine, after about 21 days of uptime, the main SSD has written about 37 TB. Process/file-level checks show Codex SQLite logs are the main continuous writer.
That extrapolates to roughly 640 TB/year. On a 1 TB SSD, that is about 640 full-drive writes per year. Some consumer SSDs are rated around 600 TBW, so this could consume roughly a full drive's warranted write endurance in less than a year.
Evidence1
A later snapshot makes the write amplification easier to see:
| metric | value |
|---|---|
current logs_2.sqlite file size |
1.2 GiB |
| current retained rows | 506,149 |
| total allocated row ids | 5,543,677,486 |
So the database currently retains only ~0.5M rows, while the SQLite AUTOINCREMENT counter has already advanced past 5.5B ids.
That is roughly a 10,000x gap between retained rows and historical inserted row ids. Even using the current ~1.2 GiB database size as a rough baseline, this points to 10TB+ scale historical log churn, before accounting for WAL, indexes, pruning, checkpoints, page rewrites, and filesystem/device-level write amplification.
Evidence2
Current retained rows in logs_2.sqlite:
| metric | value |
|---|---|
| retained rows | 681,774 |
| estimated retained log content | 1,035.6 MiB |
Level distribution:
| level | estimated MiB | byte % |
|---|---|---|
| TRACE | 732.5 | 70.7% |
| INFO | 266.5 | 25.7% |
| DEBUG | 30.6 | 3.0% |
| WARN | 5.9 | 0.6% |
Largest target+level pairs:
| target | level | estimated MiB |
|---|---|---|
codex_api::endpoint::responses_websocket |
TRACE | 527.4 |
codex_otel.log_only |
INFO | 141.2 |
codex_otel.trace_safe |
INFO | 121.2 |
log |
TRACE | 97.4 |
codex_client::transport |
TRACE | 60.1 |
codex_core::stream_events_utils |
DEBUG | 27.5 |
codex_api::sse::responses |
TRACE | 19.1 |
The top sources are mostly global TRACE logs, mirrored telemetry logs, and raw websocket/SSE payload logging. TRACE alone is about 70.7% of retained bytes. codex_otel.log_only + codex_otel.trace_safe add another 25.3%. Filtering these categories should remove roughly 96% of retained log bytes in this sample without fully disabling feedback logs.
Sanitized examples from the most frequent TRACE source: target=log
These are high-frequency retained samples. Raw websocket/SSE payload bodies are intentionally not included because they may contain private conversation content.
128,764x TRACE log: inotify event: ... mask: OPEN, name: Some("ld.so.cache")
37,982x TRACE log: inotify event: ... mask: OPEN, name: Some("locale.alias")
23,843x TRACE log: inotify event: ... mask: OPEN, name: Some("passwd")
3,639x TRACE log: <tokio-tungstenite checkout>/src/compat.rs:131 AllowStd.with_context
3,505x TRACE log: <tokio-tungstenite checkout>/src/lib.rs:245 WebSocketStream.with_context
3,362x TRACE log: <tokio-tungstenite checkout>/src/compat.rs:154 Read.read
3,356x TRACE log: <tokio-tungstenite checkout>/src/compat.rs:157 Read.with_context read -> poll_read
3,230x TRACE log: <tokio-tungstenite checkout>/src/lib.rs:294 Stream.poll_next
3,227x TRACE log: <tokio-tungstenite checkout>/src/lib.rs:304 Stream.with_context poll_next -> read()
3,213x TRACE log: inotify event: ... mask: OPEN, name: Some("nsswitch.conf")
2,001x TRACE log: WouldBlock
1,217x TRACE log: Masked: false
1,169x TRACE log: Opcode: Data(Text)
1,169x TRACE log: First: 11000001
Sanitized examples from frequent INFO sources
The dominant INFO sources are mostly repeated OpenTelemetry mirror events. IDs are redacted.
843x INFO codex_client::custom_ca:
using system root certificates because no CA override environment variable was selected ...
334x INFO codex_otel.trace_safe:
session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input" submission.id=<redacted> codex.op="user_input"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}
333x INFO codex_otel.log_only:
session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input" submission.id=<redacted> codex.op="user_input"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}
332x INFO codex_otel.log_only:
session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input_with_turn_context" submission.id=<redacted> codex.op="user_input_with_turn_context"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}
332x INFO codex_otel.trace_safe:
session_loop{thread_id=<redacted>}:submission_dispatch{otel.name="op.dispatch.user_input_with_turn_context" submission.id=<redacted> codex.op="user_input_with_turn_context"}:turn{otel.name="session_task.turn" thread.id=<redacted> ...}
Write amplification
The retained DB size hides the real write volume. In a 15-second sample:
| metric | before | after |
|---|---|---|
| retained rows | 681,774 | 681,774 |
| max row id | 5,003,347,015 | 5,003,383,226 |
About 36,211 rows were inserted in 15 seconds, while retained row count stayed flat. This suggests continuous insert-and-prune write amplification: rows are inserted, indexed, written to WAL, then pruned.
Likely cause
The SQLite feedback log sink is installed with a global TRACE default:
Targets::new().with_default(Level::TRACE)This persists all targets at TRACE level by default, including dependency/internal logs and large raw protocol payloads.
Proposed fix
Keep feedback logs enabled, but narrow what is persisted by default:
- Do not use global TRACE for the SQLite feedback log sink.
- Drop or raise thresholds for low-value dependency noise, especially
target=log,hyper_util, tokio-tungstenite internals, inotify spam, and low-level OpenTelemetry SDK logs. - Avoid persisting full raw websocket/SSE payloads by default. Store summaries instead: event kind, duration, success/error, token usage, and payload byte length.
- Avoid persisting mirrored
codex_otel.log_only/codex_otel.trace_safeevents unless they are explicitly useful for feedback debugging. - Add a global logs DB size/write cap. Per-thread caps are not enough when many threads/processes exist.
An optional escape hatch such as sqlite_logs_enabled = false would still be useful, but the main fix should be better default filtering.
Related issues and discussions
- Excessive SQLite WAL writes during streaming due to TRACE logs ignoring RUST_LOG #17320
- Codex Desktop rapidly grows logs_2.sqlite / WAL during normal active use #24275
- app-server: feedback log sqlite (logs_N.sqlite) grows unbounded — ~0.75 GB/day, no retention/rotation #26374
logs_2.sqlite-walgrows indefinitely and remains allocated after deletion because stale/suspended Codex TUI processes keep the deleted WAL open #22444- Heavy I/O activity from idle
codexprocesses. #20563 - Severe disk I/O / 100% disk active time on Windows WSL2 when using Codex extension / CLI #27020
- goals_1.sqlite write amplification: ~11 MB/s sustained writes (11 GB lifetime) on a 4 KB database #27911
- Codex Desktop becomes unusable on long active threads due to app-server/renderer memory and TRACE log churn #21134
- app-server: source /feedback logs from sqlite at trace level #12969
Activity
github-actions commentedon Jun 14, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
1996fanrui commentedon Jun 14, 2026
cc @charley-oai @jif-oai since #12969 introduced the SQLite feedback log sink at TRACE level.
I pushed a minimal branch here:
https://github.com/1996fanrui/codex/tree/codex/reduce-sqlite-feedback-log-writes
The proposed patch does not disable feedback logs. It only changes the SQLite feedback log sink from global TRACE to this default filter:
In my local sample, the retained SQLite log content was mostly TRACE plus
codex_otelINFO mirror rows, so this should drop the high-volume low-signal writes while keeping INFO+ logs for core Codex targets.I tried to open an upstream PR, but GitHub returned:
Other possible fixes if the default-filter approach is not the preferred direction:
sqlite_logs_enabled = falseor a feedback-log-specific filter setting;ZenulAbidin commentedon Jun 17, 2026
This is a very serious bug, as it can not only cause login failures on Linux boxes if the system is rebooted while the disk is full, but Codex in
/goalmode will actively delete files and folders on your disk in a vain attempt to gain disk space. So the severity of this issue needs to be increased as data loss is possible when using Codex in this way.jordanade commentedon Jun 17, 2026
WTF this is ridiculous. PLEASE have more respect for your users.
Nicolas0315 commentedon Jun 17, 2026
I prepared a PR-ready branch against current
main, but upstream PR creation is still blocked for my account with:Branch:
https://github.com/Nicolas0315/codex/tree/codex/log-db-default-filter
Compare:
https://github.com/openai/codex/compare/main...Nicolas0315:codex/log-db-default-filter?expand=1
Proposed PR title:
Proposed PR body, following the recent Codex PR format:
Why
logs_2.sqliteis currently attached with a globalTRACEfilter in both the app server and TUI. On my Windows Codex app install, that persisted high-volume websocket internals and OTel mirror records into the diagnostic SQLite log DB:.codex/logs_2.sqlite: 729,575,424 bytes.codex/logs_2.sqlite-wal: 39,255,392 bytesTRACE: 123,026 rows / 269,065,577 estimated bytescodex_api::endpoint::responses_websocket,codex_otel.log_only,codex_otel.trace_safe, andlogThis matches the write-amplification report here and the Desktop launch failure mode in #27741. The change keeps diagnostic logs enabled, but makes the persisted SQLite sink less noisy by default.
What changed
log_db::default_filter()for the persistent SQLite log sink.INFO+by default instead ofTRACE.WARN+:codex_otel.log_onlycodex_otel.trace_safehyper_util::client::legacy::*logopentelemetry_sdkINFOlogs and dropped low-value OTel/mirror logs.Validation
git diff --checkjust test -p codex-state sqlite_sink_default_filter_drops_low_value_logsjust test -p codex-tui sqlite_sink_default_filter_drops_low_value_logscompiled the package, then exited with no matching tests.just fmtcompleted Rust/Python formatting, then failed on Windows in the Bazel buildifier step with[WinError 2].just test -p codex-app-serverrun compiled and executed tests but timed out after reporting831 passed, 4 failed, 6 skipped; the visible failures appeared unrelated to this log filtering change, including a skills fixture warning count mismatch.ZenulAbidin commentedon Jun 18, 2026
Here I have made two scripts which can serve as temporary mitigations for the WAL issue until OpenAI fixes it.
The first script trims the WAL while Codex processes are running:
To run as a cron job:
Note that the main table is trimmed, but there might be other tables which still accumulate data over a much slower period of time, requiring the script below.
The second script deletes the log and WAL files and kills all running Codex processes in order to immediately free disk space:
Gerry9000 commentedon Jun 19, 2026
plz fix
Necmttn commentedon Jun 21, 2026
The log DB should have an explicit byte budget and retention policy. Durable usage/diagnostic events can stay queryable, while websocket/SSE TRACE payloads need sampling, level caps, and rotation independent of the state DB. A startup receipt with effective log level, DB byte cap, WAL checkpoint policy, and dropped-row count would make write pressure visible before SSD wear becomes the signal.
Generated with ax.
Nicolas0315 commentedon Jun 21, 2026
Additional native Windows check against current
main:63f009e9dad2e70454b7ed6434d8aa28dfb52b51Current source still attaches the persistent SQLite log layer with a global TRACE filter in both app-server and TUI:
codex-rs/app-server/src/lib.rslog_db::start.with_filter(Targets::new().with_default(Level::TRACE))codex-rs/tui/src/lib.rslog_db::start.with_filter(Targets::new().with_default(Level::TRACE))I also checked the current log sink implementation:
codex-rs/state/src/log_db.rsfeedback_log_body, queued, then inserted by a background task.opentelemetry_sdkTRACE/DEBUG, which helps one specific noisy target.codex-rs/state/src/runtime/logs.rsThat means DB size or retained row count can stay stable while insert/prune churn continues.
Local aggregate-only measurement from
.codex\logs_2.sqliteover 15 seconds, without reading log bodies:MAX(id)before: 191,131,613MAX(id)after: 191,131,862MAX(id)delta: 249 rows inserted in 15sTop retained target+level groups in this Windows sample were still dominated by the same categories already reported here:
codex_api::endpoint::responses_websocketTRACEcodex_otel.log_onlyINFOcodex_otel.trace_safeINFOcodex_core::stream_events_utilsINFOcodex_client::transportTRACErmcp::serviceTRACEcodex_api::sse::responsesTRACElogTRACEThis reinforces the earlier point: visible DB growth is not a reliable proxy for write pressure. Even with a stable retained row count, the append-and-prune path is still doing work. The low-risk patch shape still looks like filtering before formatting/queueing/inserting, not only pruning after insert.
I also rebuilt a narrow local candidate patch against the current
origin/mainhead:f774455(code-mode: linearize cell terminal state (#29286))codex_state::log_db::default_filter()Targets::new().with_default(Level::TRACE)WARN+INFO+for core Codex targets (codex_api,codex_app_server,codex_core,codex_mcp,codex_state,codex_tui)WARN+(codex_otel.log_only,codex_otel.trace_safe,hyper_util,log,opentelemetry_sdk)Local validation:
cargo test -p codex-state sqlite_sink_default_filter_drops_low_value_logs-> 1 passedcargo test -p codex-state log_db-> 7 passedcargo check -p codex-app-server -p codex-tui-> passedcargo fmt --check-> passed; stable rustfmt emitted repo-config warnings thatimports_granularityis nightly-onlygit diff --check-> passedPer
docs/contributing.md, I am not opening a PR unless invited, but this latest local patch is PR-shaped if maintainers want this direction.cooltzy commentedon Jun 22, 2026
I hope the developers see this issue.
kernbug commentedon Jun 22, 2026
Shame on such low software quality with such high memory prices and SSD cost...
153 remaining items
rodalpho commentedon Jul 12, 2026
Which version has it fixed? Is it in the desktop app?
etraut-openai commentedon Jul 12, 2026
Our series of fixes for this problem are in the latest shipping desktop app, CLI, and IDE extension.
darlingm commentedon Jul 12, 2026
Not quite yet. 0.144.1 does not have #31789, #31790, #31791, and #31792. 0.145.0-alpha.4 includes them, so 0.145.0 likely will too.
Also, please see #32496. A different root cause is causing about 1.3 TB/year in unnecessary writes, according to the numbers in the issue. (Much less than was on this issue.) Codex is continually rewriting
models_cache.jsonandscope_v3.json.Would it be possible for OpenAI to add a regression test that ensures: (1) Codex during idle doesn't make more than a very minimal amount of I/O; and (2) Codex during usage doesn't absurdly increase its I/O? It might be best to split 2, into one version that fakes the model to test the harness in isolation, and one end-to-end test that makes sure new model/harness combinations don't come up with something that bumps it significantly. The latter could of course trip from random model behavior. If OpenAI logged and compared the values, they would not only see spikes but also worrisome trends in I/O usage that could be investigated.
While at it, testing CPU and GPU usage similarly would be a big help. There are several issues about crazy high usage, often tracked down to animations being unnecessarily repeatedly updated a crazy number of times a second.
gustavopch commentedon Jul 12, 2026
@etraut-openai Codex has been tracking this issue for me. I asked its opinion (GPT 5.6 Sol High), and it doesn't think this is fully solved:
I retested the SQLite logging behavior on macOS. The upstream fixes have substantially reduced the original churn, but the current desktop build still persists noisy TRACE/DEBUG activity at a much higher rate than when the local filter is enabled.
The mitigation is a local SQLite trigger that rejects TRACE/DEBUG records and known noisy targets. It does not block INFO, WARN, or ERROR logging.
This shows two things:
During the current-build test without mitigation, 305 IDs were allocated while only 9 additional rows remained. That is consistent with continuing insert-and-prune activity.
The newly retained records were dominated by:
No log bodies were read during these measurements.
Is this residual TRACE/DEBUG churn considered expected after the fixes? If not, is desktop app 26.707.51957 missing part of the fix, and which desktop or bundled CLI version should contain it?
0xdevalias commentedon Jul 13, 2026
@gustavopch It seems like you're still using
0.144.0-alpha.4in those tests; whereas an earlier comment from @darlingm suggests that there are more fixes that will land in0.145.0-alpha.4+, that sound like they should address some if not all of those:daryll-swer commentedon Jul 13, 2026
So are we getting some refunds/limit resets or ANY compensation for killing our SSDs? For weeks or nah?
gustavopch commentedon Jul 13, 2026
@0xdevalias Ah, thanks for pointing it out. I just updated from
26.707.51957to26.707.61608and the bundled Codex CLI is still0.144.0-alpha.4. I guess we'll have to wait a few more days.HGT158 commentedon Jul 20, 2026
So has this issue been fixed in the latest version of the Codex desktop app?
feat: Codex CLIのログ用SQLiteをtmpfsへ逃がしSSD摩耗を防ぐ
fix(codex-native): share plugins/cache into per-session homes (#3401)
Publish codex-fork 0.145 mainline (#15)
fix: sweep per-session tmp files by age in session-start
feat(skill): codex-log-monitor v0.1.0 — logs_2.sqlite 增长监控/block trig…
fix(codex-native): share plugins/cache into per-session homes (omnige…
fix(codex-native): share plugins/cache into per-session homes (omnige…