Multi-terminal codex CLI freezes due to SQLite lock contention with no BUSY retry

[solosvip]

Summary

Running multiple codex CLI instances against the same $CODEX_HOME causes
TUI freezes: input echo lags by seconds, and streamed assistant output can
deadlock entirely (only ctrl-C recovers). The root cause is contention on the
shared state_5.sqlite and logs_2.sqlite, combined with the absence of
SQLITE_BUSY retry logic in codex-rs/state/src/runtime/logs.rs.

Environment

• codex-cli 0.125.0
• macOS 26.3.1, Apple Silicon
• Single $CODEX_HOME shared across multiple terminals (default usage)

Reproduction

1. Open 2+ terminals, all running codex against the same $CODEX_HOME.
2. In each terminal, send a prompt at roughly the same instant.
3. Observe: TUI input lag, streamed output stalling mid-response, occasional
   permanent freeze requiring kill.

Evidence

• logs_2.sqlite (the OTel trace sink) grew to 249 MB / 45,000+ rows in
  ~1.5 days of normal use; every SSE chunk emits a TRACE row.
• state_5.sqlite is in WAL mode with busy_timeout = 5s (set in
  state/src/runtime.rs), but no BUSY retry is implemented in
  state/src/runtime/logs.rs::insert_logs. On contention the call surfaces
  the error to upstream stream-handling code, which appears to drop the
  channel and leave the TUI waiting forever.
• Truncating logs_2.sqlite (DELETE FROM logs; VACUUM;) immediately
  reduces the freeze frequency by ~80% — confirming the OTel sink is the
  dominant contender.
• Even after truncation, simultaneous prompts across terminals still
  produce ~1-2s input lag, indicating residual contention on state_5.sqlite
  (threads / agent_jobs writes).

Suggested fixes

1. Add BUSY retry to insert_logs (and other sqlx writes against
   logs_2.sqlite / state_5.sqlite): wrap in a bounded retry loop with
   exponential backoff before propagating the error to TUI.
2. Increase busy_timeout from 5 s to 30 s (or make it configurable).
   5 s is too short for 200 MB+ WAL files.
3. Per-process OTel sink: write logs_2.sqlite to a per-PID file
   (e.g. logs_2.<pid>.sqlite) and merge offline. OTel traces are
   write-only and rarely read interactively, so sharding is safe.
4. Bound the OTel TRACE level by default: writing every SSE chunk at
   TRACE inflates the DB by ~150 MB/day. Default to INFO and let users
   opt into TRACE.
5. Treat sink-write failure as non-fatal for the stream pipeline:
   even if the log insert fails, the model stream consumer should not
   abandon the channel — log the error and keep reading SSE.

Bonus issue (related, may want a separate ticket)

With $CODEX_HOME set to a non-default directory, codex appears to load
hooks from both $CODEX_HOME/hooks.json and ~/.codex/hooks.json,
firing each Stop hook twice. Confirmed by paired log entries with identical
timestamps in the hook script's own log file.

[etraut-openai]

Thanks for the report, but I don't think your analysis is correct.

Current code does use shared state_5.sqlite / logs_2.sqlite, WAL mode, and a 5s busy_timeout, and insert_logs does not have an explicit retry loop. It’s also plausible that the persistent log DB can grow quickly because the TUI installs the SQLite log layer at TRACE and SSE processing logs each stream event.

That said, the local SQLite log sink is intentionally asynchronous: tracing events are sent through a bounded background queue, and the background flush currently ignores insert_logs failures. So a BUSY error in logs_2.sqlite should not directly propagate into stream handling or drop the model stream/TUI channel. It may drop log rows or add background I/O pressure, but there is no direct “log insert failure aborts stream consumer” path in current code.

[mrairdon-midmark]

Adding a Windows Desktop data point that looks like the same root cause, with a source-level pointer.

Environment:

• Codex Desktop: 26.429.2026.0
• @openai/codex: 0.128.0
• Platform: Windows
• Auth: ChatGPT account

Local evidence:

• %USERPROFILE%\.codex\logs_2.sqlite grew to about 1.01 GB
• logs table had 318,757 rows
• During the slow UI period, the bundled app-server process averaged about 26.6% CPU and had disk read/write bursts
• Recent slow SQL warnings included:
  • PRAGMA wal_checkpoint(TRUNCATE) taking 5.58s during thread/list
  • slow batched INSERT INTO logs (...)
• Clearing the log SQLite table dramatically improved Codex Desktop app performance.

The largest local log contributors were:

• codex_api::endpoint::responses_websocket: ~83k rows / ~365 MB body text
• codex_otel.log_only: ~85k rows / ~109 MB body text
• codex_otel.trace_safe: ~84k rows / ~100 MB body text

Source-level signal from current upstream:

• The app-server installs the SQLite log DB layer with Targets::new().with_default(Level::TRACE):
  

  codex/codex-rs/app-server/src/lib.rs

  Lines 581 to 584 in f50c02d

  	let log_db = state_db.clone().map(log_db::start);
  	let log_db_layer = log_db
  	.clone()
  	.map(|layer| layer.with_filter(Targets::new().with_default(Level::TRACE)));

• The WebSocket client logs full websocket request payloads at TRACE:
  

  codex/codex-rs/codex-api/src/endpoint/responses_websocket.rs

  Lines 548 to 556 in f50c02d

  	let request_text = match serde_json::to_string(&request_body) {
  	Ok(text) => text,
  	Err(err) => {
  	return Err(ApiError::Stream(format!(
  	"failed to encode websocket request: {err}"
  	)));
  	}
  	};
  	trace!("websocket request: {request_text}");

• The WebSocket client also logs full websocket event payloads at TRACE:
  

  codex/codex-rs/codex-api/src/endpoint/responses_websocket.rs

  Lines 597 to 599 in f50c02d

  	match message {
  	Message::Text(text) => {
  	trace!("websocket event: {text}");

So this does not appear to be only multi-terminal contention. In Desktop, normal app-server use can persist very large TRACE rows, including full responses_websocket request/event payloads, into logs_2.sqlite. Once the DB is large enough, checkpoints and inserts become visible as CPU/disk bursts and UI slowness.

Workaround that helped locally: quit Codex, then clear/checkpoint/vacuum only logs_2.sqlite with:

uv run python -c "import sqlite3, os; p=os.path.join(os.environ['USERPROFILE'],'.codex','logs_2.sqlite'); con=sqlite3.connect(p, timeout=60); con.execute('PRAGMA busy_timeout=60000'); con.execute('DELETE FROM logs'); con.commit(); con.execute('PRAGMA wal_checkpoint(TRUNCATE)'); con.execute('VACUUM'); con.close()"

Do not run that against state_5.sqlite; this workaround only clears local log rows.

[mrairdon-midmark]

@etraut-openai yeah, that makes sense for the narrow failure path: I agree a BUSY or failed insert_logs probably should not directly abort the stream if the log sink is queuing and dropping insert errors.

I’m wondering if the Desktop slowdown could still come from the log sink indirectly, though, rather than through error propagation:

responses websocket event
  -> trace!(full payload)
  -> LogDbLayer::on_event on emitting task
  -> format span context + payload
  -> queue for background insert
       -> INSERT / checkpoint / WAL work on logs_2.sqlite

Possible latency sources:
  - CPU/allocation before the queue boundary
  - CPU/disk/SQLite pressure from the background writer
  - slower app-server RPCs like thread/list
  - Desktop UI feels unresponsive

So I’m not thinking “log insert failure aborts stream.” More like:

• the SQLite write is async, but event formatting happens before the queue boundary
• the trace site is on a hot websocket streaming path and includes full request/event payloads
• the async writer still shares the same app-server process/runtime/disk
• once logs_2.sqlite is large, checkpoint/WAL work may become visible latency; I saw PRAGMA wal_checkpoint(TRUNCATE) take 5.58s under a thread/list span

Does that path sound plausible, or is there something about the app-server/runtime split that should prevent this from affecting Desktop RPC latency? The part that keeps me suspicious is that clearing only the logs table plus checkpoint/vacuum was followed by Desktop going from very slow back to normal immediately.

[drony]

@etraut-openai yeah, that makes sense for the narrow failure path: I agree a BUSY or failed insert_logs probably should not directly abort the stream if the log sink is queuing and dropping insert errors.

I’m wondering if the Desktop slowdown could still come from the log sink indirectly, though, rather than through error propagation:

responses websocket event
  -> trace!(full payload)
  -> LogDbLayer::on_event on emitting task
  -> format span context + payload
  -> queue for background insert
       -> INSERT / checkpoint / WAL work on logs_2.sqlite

Possible latency sources:
  - CPU/allocation before the queue boundary
  - CPU/disk/SQLite pressure from the background writer
  - slower app-server RPCs like thread/list
  - Desktop UI feels unresponsive

So I’m not thinking “log insert failure aborts stream.” More like:

• the SQLite write is async, but event formatting happens before the queue boundary
• the trace site is on a hot websocket streaming path and includes full request/event payloads
• the async writer still shares the same app-server process/runtime/disk
• once logs_2.sqlite is large, checkpoint/WAL work may become visible latency; I saw PRAGMA wal_checkpoint(TRUNCATE) take 5.58s under a thread/list span

Does that path sound plausible, or is there something about the app-server/runtime split that should prevent this from affecting Desktop RPC latency? The part that keeps me suspicious is that clearing only the logs table plus checkpoint/vacuum was followed by Desktop going from very slow back to normal immediately.

Thanks, it worked for me too.