Skip to content

Windows Codex 26.623.5546.0 still persists high-frequency TRACE logs to logs_2.sqlite WAL #30405

Open
Open
Windows Codex 26.623.5546.0 still persists high-frequency TRACE logs to logs_2.sqlite WAL#30405
Labels
appIssues related to the Codex desktop appapp-serverIssues involving app server protocol or interfacesbugSomething isn't workingperformancewindows-osIssues related to Codex on Windows systems

Description

@fffire-oss
fffire-oss
opened on Jun 28, 2026

Summary

On Windows, Codex Desktop 26.623.5546.0 still persists high-frequency TRACE logs into the local SQLite feedback log database (logs_2.sqlite) and its WAL during normal active use.

This appears related to existing reports such as #17320, #28224, #29532, #29832, and #30236, but I am filing this with fresh Windows data from 26.623.5546.0.

Environment

  • OS: Microsoft Windows NT 10.0.26200.0, x64
  • Codex app package: OpenAI.Codex_26.623.5546.0_x64__2p2nqsd0c76g0
  • App-server command: codex.exe app-server --analytics-default-enabled
  • Observed command runner: codex-command-runner-0.142.3.exe
  • Files observed:
    • C:\Users\<user>\.codex\logs_2.sqlite
    • C:\Users\<user>\.codex\logs_2.sqlite-wal
    • C:\Users\<user>\.codex\state_5.sqlite-wal
  • Current shell environment had RUST_LOG=warn; no user/machine RUST_LOG override was found.

Observed behavior

A 30-second local sample during normal Codex use showed continuous insert/prune churn in logs_2.sqlite:

logs.max(id):   21,156,392 -> 21,181,109
Delta ids:      +24,717 in 30 seconds
Approx rate:    ~824 inserted log ids/sec
Retained rows:  44,757 -> 44,852
Delta rows:     +95

Recent persisted rows after the sample were entirely TRACE-level:

Recent 1000 rows by level:
TRACE 1000

Recent 1000 rows by target:
TRACE log                                  949
TRACE codex_app_server::outgoing_message   51

File sizes and mtimes at the same point:

logs_2.sqlite       147,492,864 bytes
logs_2.sqlite-wal    25,206,192 bytes
state_5.sqlite-wal    4,243,632 bytes

A separate 30-second mtime sample showed logs_2.sqlite-wal, logs_2.sqlite, and state_5.sqlite-wal modification times advancing repeatedly while the .sandbox text log did not continue growing. This suggests the ongoing write pressure is mostly SQLite/WAL churn rather than simple text-log append growth.

Representative TRACE bodies were not included because they may contain private local/session data. Sanitized inspection showed low-level websocket/tokio/tungstenite style entries such as poll_read, poll_next, WouldBlock, frame metadata, and large frame payload logging.

Disk write / SSD endurance concern

A Windows Performance Counter sample over the same 30-second window reported the codex process at approximately:

Average process IO Write Bytes/sec: ~1,346,401 B/s (~1.35 MB/s)
Max process IO Write Bytes/sec:     ~15,324,126 B/s (~15.3 MB/s)

I understand Windows process IO counters are not identical to physical NAND writes and may include non-disk IO. The physical disk counter is also system-wide, not Codex-only. Still, the SQLite evidence above confirms that a large part of the churn is local persistence.

If the measured ~1.35 MB/s write rate were sustained, it would be approximately:

~116 GB/day
~42.5 TB/year

That is a meaningful fraction of common consumer SSD endurance ratings:

150 TBW drive:  ~28% of warranty TBW/year
300 TBW drive:  ~14% of warranty TBW/year
600 TBW drive:   ~7% of warranty TBW/year
1200 TBW drive:  ~3.5% of warranty TBW/year

Short bursts at the observed peak are not equivalent to sustained writes, but the issue is concerning because the churn is mostly diagnostic TRACE data and can recur throughout long Codex sessions.

Expected behavior

Production Codex Desktop builds should not persist high-frequency frame-level / dependency-level TRACE logs into logs_2.sqlite by default, especially when RUST_LOG=warn is in effect.

Expected improvements:

  • The SQLite log sink should honor the same effective log-level filter as the runtime logger, or have its own documented configuration.
  • Low-value TRACE target=log and websocket/tungstenite frame logs should be suppressed, sampled, summarized, or bounded.
  • Local diagnostic logs should have clear size/write caps, rotation, and WAL checkpoint/truncation behavior.
  • Users should have a documented option to disable or reduce persistent local diagnostic logging without disabling durable thread/state data.

Privacy note

I am not attaching logs_2.sqlite or raw feedback_log_body values because they may contain private session text, local paths, tool output, or protocol payload fragments.

Activity

  1. github-actions
    added
    bugSomething isn't working
    appIssues related to the Codex desktop app
    windows-osIssues related to Codex on Windows systems
    app-serverIssues involving app server protocol or interfaces
    performance
    on Jun 28, 2026

  2. github-actions commented on Jun 28, 2026

    @github-actions
    github-actionsBot
    on Jun 28, 2026 – with GitHub Actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  3. Nicolas0315 commented on Jun 28, 2026

    @Nicolas0315
    Nicolas0315
    on Jun 28, 2026

    Additional 2026-06-29 JST sample after reducing active session JSONL load with official codex archive:

    • Codex Desktop version: 26.623.5546.0
    • Codex CLI: 0.142.3
    • ~/.codex/config.toml already has:
      • [analytics] enabled = false
      • [feedback] enabled = false
      • [otel] exporter = "none"
      • [otel] metrics_exporter = "none"
      • [otel] trace_exporter = "none"
      • [otel] log_user_prompt = false
    • 10s metadata-only SQLite sample, without reading feedback_log_body:
      • logs.id delta: 1,037 rows
      • WAL size delta: 0 bytes in this short window
      • Top targets:
        • TRACE log: 352 rows, estimated 352,063 bytes
        • DEBUG log: 140 rows, estimated 65,131 bytes
        • TRACE hyper_util::client::legacy::pool: 120 rows
        • TRACE hyper_util::client::legacy::client: 78 rows
        • TRACE codex_app_server::outgoing_message: 53 rows
    • 15s process CPU sample after session archive, excluding benchmark parse work:
      • total Codex/ssh/node_repl matched processes: 8.11% of one core
      • top Codex processes: 5.62%, 1.46%, 0.62%

    So session JSONL load can be mitigated separately, but high-frequency TRACE/DEBUG persistence remains observable even with analytics/feedback/otel disabled in config.

  4. Nicolas0315 commented on Jun 28, 2026

    @Nicolas0315
    Nicolas0315
    on Jun 28, 2026

    Follow-up from the same Windows Codex Desktop profile (Codex Desktop 26.623.5546.0, CLI 0.142.3; 2026-06-29 03:40 JST):

    Config/docs check:

    • local ~/.codex/config.toml has [analytics] enabled=false, [feedback] enabled=false, [otel] exporter="none", metrics_exporter="none", trace_exporter="none", log_user_prompt=false
    • current shell has RUST_LOG=warn; HKCU persistent user env has no RUST_LOG
    • fresh local Codex manual documents [analytics], [feedback], [otel], and RUST_LOG, but I did not find a documented knob specifically disabling local logs_2.sqlite TRACE persistence/retention beyond those settings
    • Desktop launches app-server as resources\codex.exe app-server --analytics-default-enabled

    Read-only logs_2.sqlite aggregate, no log body/prompt fields read:

    • DB size: 102.5 MiB, rows: 26,317
    • all-time levels: TRACE 19,243; DEBUG 3,528; INFO 3,130; WARN 311; ERROR 100
    • last 5 minutes: 1,909 rows
    • last 5 minute top pairs: TRACE log 973; TRACE codex_mcp::connection_manager 494; TRACE codex_api::sse::responses 117; DEBUG codex_core::stream_events_utils 89

    So TRACE persistence is still happening in normal Desktop usage even with analytics/feedback/OTel disabled in config. This continues to push the log DB past the local soft cap and adds SQLite/WAL pressure while the app is open.

  5. Nicolas0315 commented on Jun 28, 2026

    @Nicolas0315
    Nicolas0315
    on Jun 28, 2026

    Additional local mitigation to test on next Desktop restart (2026-06-29 04:19 JST):

    Even after stopping idle MCP/plugin/node_repl helper processes, a 30s metadata-only sample still showed logs_2.sqlite writes at about 4.9 rows/s, led by TRACE log rows. The current shell had RUST_LOG=warn, but Codex Desktop launches app-server via Codex.exe, so the app-server likely did not inherit that transient shell env.

    I persisted user-scope RUST_LOG=warn under HKCU:\Environment with restore metadata. Previous user value was absent. This should only take effect for future Codex Desktop/app-server launches. I will treat this as unverified until after a Desktop restart and another logs_2.sqlite write-rate sample.

    Rollback is removing user RUST_LOG. If TRACE rows continue after restart, the local SQLite TRACE persistence is not controlled by RUST_LOG and needs product-side filtering/retention.

  6. martinrodriguezr93-cloud commented on Jun 28, 2026

    @martinrodriguezr93-cloud
    martinrodriguezr93-cloud
    on Jun 28, 2026

    Additional affected Windows machine data point from Codex Desktop 26.623.5546.0.

    Environment:

    • OS: Windows, x64
    • Codex Desktop package: OpenAI.Codex_26.623.5546.0_x64__2p2nqsd0c76g0
    • Microsoft Store / winget reports no newer Store package available
    • Active Desktop app-server command: resources\codex.exe app-server --analytics-default-enabled
    • Local command runner present: codex-command-runner-0.142.3.exe
    • I also downloaded and verified the official GitHub prerelease CLI rust-v0.143.0-alpha.29 separately:
      • codex-cli 0.143.0-alpha.29
      • SHA256 codex.exe: EEA11264DFA89B7E6C4ECF45C0DC159D638929218DAF035D3EC31485CAC1555A
      • This does not update the Microsoft Store Desktop app-server path, so Desktop remains affected.

    SSD / SMART context:

    • Disk: WD_BLACK SN850X 1000GB NVMe
    • SMART Data Units Written: 5,583,445
    • Converted host writes: about 2.859 TB decimal / 2.60 TiB
    • Power on hours: 225
    • SMART Percentage Used: 0
    • Media errors: 0
    • The disk is new and used mostly for Codex work, so avoidable diagnostic write churn is meaningful even though lifetime wear is still low.

    Before local mitigation, metadata-only SQLite samples against C:\Users\<user>\.codex\logs_2.sqlite showed continuing insert/prune churn:

    DB size: ~2.13 GiB
logs_2.sqlite-wal: ~95.23 MiB
retained rows: ~415k
max(id): >328M

20s sample:
delta_count: +51
delta_max_id: +8,442

30s sample:
delta_count: -129
delta_max_id: +4,868
ids_per_sec: ~162.3

60s sample:
delta_count: +1
delta_max_id: +8,976
ids_per_sec: ~149.6

    Recent retained rows were still dominated by TRACE, with top pairs including:

    TRACE log
TRACE codex_api::sse::responses
TRACE codex_mcp::connection_manager
TRACE codex_core::client

    Windows process I/O counters during the affected run also showed large cumulative Codex writes since process start:

    codex.exe app-server process: ~219 GB written since process start
Codex matched processes total: ~295 GB written since process start

    I understand process I/O counters are not the same as physical NAND writes, but the SQLite metadata confirms real local persistence churn.

    Local workaround applied:

    • Backed up logs_2.sqlite, logs_2.sqlite-wal, and logs_2.sqlite-shm
    • Added a SQLite trigger:
    CREATE TRIGGER IF NOT EXISTS codex_block_logs_insert_ssd_mitigation_20260628
BEFORE INSERT ON logs
BEGIN
  SELECT RAISE(IGNORE);
END;
    • Ran PRAGMA wal_checkpoint(TRUNCATE)

    After workaround:

    60s sample:
delta_count: 0
delta_max_id: 0
delta_estimated_mb: 0
wal_delta_mb: 0

logs_2.sqlite-wal: 95.23 MB -> 0 MB

    Conclusion:

    Codex Desktop 26.623.5546.0 on Windows still appears to persist or attempt high-frequency diagnostic TRACE rows into logs_2.sqlite during ordinary Desktop use. The local trigger stops SQLite inserts, but this is only a defensive workaround. The product still needs an upstream fix: production Desktop should not persist high-frequency TRACE/DEBUG transport/dependency logs by default, and there should be a documented user-facing kill switch or bounded persistent-log policy for local diagnostic SQLite logs.

    Privacy note: I did not attach logs_2.sqlite or raw feedback_log_body values because they may contain private prompts, local paths, tool output, or protocol payload fragments.

  7. martinrodriguezr93-cloud commented on Jun 28, 2026

    @martinrodriguezr93-cloud
    martinrodriguezr93-cloud
    on Jun 28, 2026

    Follow-up from the same Windows machine after testing the official alpha binary:

  8. lvchongjun123-bot commented on Sep 6, 2026

    @lvchongjun123-bot
    lvchongjun123-bot
    on Sep 6, 2026

    Follow-up: codex_app_server::outgoing_message still floods SQLite logs

    Windows desktop 0.150.0-alpha.12.2 still persists high-frequency TRACE from
    codex_app_server::outgoing_message into logs_2.sqlite.

    Measured on 2026-09-06:

    • 996 rows for codex_app_server::outgoing_message in a 21-second sample
    • Current retained rows: 2,540; MAX(id): 1,758,524
    • logs_2.sqlite: about 101 MB, WAL about 4 KB
    • The SQLite sink defaults to TRACE; RUST_LOG=warn does not affect it
    • A local INSERT trigger stops database growth, but event formatting and queue
      work still occur, so typing/UI latency persists during streaming deltas

    The missing change is in codex-rs/state/src/log_db.rs default_filter():

    .with_target(
    "codex_app_server::outgoing_message",
    LevelFilter::DEBUG,
)

    Prepared patch:
    https://gist.github.com/lvchongjun123-bot/66602393eda44001a6e426ff9132b246

    Related Windows directory-scan symptom:
    #33737

  9. terryclarkeperth1 commented on Sep 22, 2026

    @terryclarkeperth1
    terryclarkeperth1
    on Sep 22, 2026

    I reproduced persistent SQLite TRACE churn on Windows Desktop and prepared a focused source patch: commit 113250b.

    In a one-hour live sample, logs_2.sqlite recorded 1,743 codex_app_server::outgoing_message, 273 codex_core::session::world_state, and 247 codex_core::session::turn TRACE messages. The current codex-rs/state/src/log_db.rs filters other noisy targets, but not these three. The patch adds DEBUG sink thresholds for them, keeping DEBUG and higher messages. The added regression test checks both sides of that threshold.

    Verification: cargo +stable test -p codex-state passed (199 unit tests and 1 doc test), cargo +stable fmt --all -- --check passed, and git diff --check passed. I attempted an upstream PR, but GitHub rejected it; the current contribution guide says external PRs are not accepted. The linked commit is ready for the Codex maintainers to apply or adapt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    appIssues related to the Codex desktop appapp-serverIssues involving app server protocol or interfacesbugSomething isn't workingperformancewindows-osIssues related to Codex on Windows systems

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions