Skip to content

Heavy I/O activity from idle codex processes. #20563

Description

@emcd

What version of Codex CLI is running?

0.128.0

What subscription do you have?

Plus

Which model were you using?

gpt-5.5

What platform is your computer?

Linux 6.17.0-1014-nvidia aarch64 aarch64

What terminal emulator and version are you using (if applicable)?

No response

What issue are you seeing?

Completely idle codex processes dominating I/O on the host.

See attached screenshot, for example:

Image

During diagnosis of the issue, GPT 5.5 noticed that all of the processes are writing a lot to a codex-tui.log file, but even its large size does not account for the amount of I/O. Does not seem to be network-related however. Diagnostics included in "Additional information" section below.

What steps can reproduce the bug?

Observe process I/O with htop or similar tools.

What is the expected behavior?

Idle codex processes should not be bombarding the system with writes at 100-200 KB/s.

Additional information

Diagnostics from GPT 5.5 via permissions escalation to me for calls needing special access to /proc or Linux capabilities not granted by the sandbox:


Codex CLI Idle Disk I/O Diagnostics

Additional Information

Observed on Codex CLI 0.128.0 on Linux/aarch64:

  • Package: @openai/codex@0.128.0
  • Native binary: @openai/codex-linux-arm64, aarch64-unknown-linux-musl
  • Node: v24.14.1
  • npm: 11.11.0
  • OS: Ubuntu/Linux 6.17.0-1014-nvidia, aarch64
  • Filesystem: ext4

I had seven long-running Codex TUI sessions open in a tmux-managed
infrastructure group. They were visually idle at prompts, with no active user
interaction and no known unfinished tasks. htop showed high disk write
activity from all Codex processes, so I gathered non-interactive diagnostics.

Per-process Disk I/O

Using pidstat -d -p <codex-pids> 1 10, idle Codex processes showed recurring
write activity, typically hundreds of KB/s per process. One sample average over
10 seconds:

PID 1066292: 149.20 kB_wr/s
PID 1066351: 221.20 kB_wr/s
PID 1066373: 144.80 kB_wr/s
PID 1066443: 150.00 kB_wr/s
PID 1066502: 146.40 kB_wr/s
PID 1066533: 1077.60 kB_wr/s
PID 1066534: 144.40 kB_wr/s

A direct /proc/<pid>/io before/after sample over 10 seconds showed idle
sibling processes writing roughly 1.4-3.2 MB each:

PID 1066292: +1,650,688 write_bytes
PID 1066351: +2,977,792 write_bytes
PID 1066373: +1,474,560 write_bytes
PID 1066443: +3,162,112 write_bytes
PID 1066502: +1,605,632 write_bytes
PID 1066534: +1,531,904 write_bytes

The currently active diagnostic session wrote more, as expected, but the
important observation is that the otherwise idle sessions also showed sustained
write activity.

Files Open by Idle Codex Processes

lsof and /proc/<pid>/fd showed each Codex process with open handles to
shared Codex state/log databases and WAL files, including:

~/.config/codex/state_5.sqlite
~/.config/codex/state_5.sqlite-wal
~/.config/codex/state_5.sqlite-shm
~/.config/codex/logs_2.sqlite
~/.config/codex/logs_2.sqlite-wal
~/.config/codex/logs_2.sqlite-shm
~/.config/codex/log/codex-tui.log
~/.config/codex/sessions/.../rollout-<session-id>.jsonl

Current file sizes during diagnosis:

codex-tui.log:        ~2.89 GB
logs_2.sqlite:        ~168.6 MB
logs_2.sqlite-wal:      ~7.3 MB
state_5.sqlite:        ~307 KB
state_5.sqlite-wal:     ~4.1 MB

File Growth Versus Write Counters

A 15-second file-size sample showed that idle session transcript files were not
growing:

Idle session JSONL files: no growth
state_5.sqlite-wal: no size growth
logs_2.sqlite-wal: no size growth
codex-tui.log: +83,379 bytes
active session JSONL: +2,334 bytes

This suggests much of the idle write activity is likely rewriting/flushing
existing SQLite/WAL pages or shared state/log database files rather than
appending to the per-session JSONL transcript.

Network Observations

lsof -iTCP showed established HTTPS connections for most Codex processes to
172.64.155.209:443.

However, at least one idle process in the sample did not show an established TCP
connection at that moment and still wrote about 2.98 MB over 10 seconds. So
the idle disk write activity does not appear to require an active model response
stream.

TUI Log Verbosity During Active Turns

During active work, codex-tui.log appears extremely verbose. The last 5000
lines contained:

websocket_event: 4393
log_only:        2249
trace_safe:      2235

Most common websocket event kinds:

2772 response.function_call_arguments.delta
1191 response.output_text.delta
76   response.output_item.done
74   response.output_item.added
52   response.completed

The log appears to record both codex_otel.log_only and
codex_otel.trace_safe entries for many websocket delta events. The log_only
entries also included account metadata such as account id and email address, so
raw log excerpts require redaction before sharing.

Working Hypothesis

There may be two related sources of disk I/O:

  1. Idle Codex TUI processes appear to perform periodic local persistence or
    telemetry/state writes, likely involving the shared SQLite databases and WAL
    files under ~/.config/codex.
  2. Active turns additionally generate very large codex-tui.log output due to
    per-delta websocket telemetry logging.

The idle case is the surprising part: multiple Codex TUI sessions sitting at
prompts still produced sustained disk writes even though their per-session JSONL
transcript files did not grow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIbugSomething isn't workingperformance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions