Codex app-server can grow to 30-40 GB when local rollout history is huge

Summary

After updating/restarting Codex Desktop, the codex app-server process grew to tens of GB on a 16 GB macOS machine, causing extreme swap usage and memory pressure. Local evidence points to a pathological persisted rollout/session history interacting with app-server thread listing/resume/read paths.

This report is intentionally sanitized: I am not attaching private transcript contents, only process metrics, file sizes, record counts, crash UI, and public repo code paths.

Environment

Codex Desktop: 26.616.51431 build 4212
Codex app Chromium framework: 149.0.7827.115
macOS: 26.3 (25.3.0), arm64
Physical memory: 16 GB
Repo inspected: openai/codex at 5f129a4703ceaf843e1cb6996bbf4a4f21225198

Observed impact

Activity Monitor initially showed codex at ~22.4 GB with ~15.34 GB swap.
A later screenshot showed the same codex process at ~39.05 GB with ~32.06 GB swap.
A live footprint/vmmap capture while the process was bad showed:
- codex [690] physical footprint: ~30 GB
- phys_footprint_peak: ~30 GB
- MALLOC_SMALL: ~30 GB dirty
- vmmap: writable regions total ~30.9 GB; ~28.9 GB swapped out
OpenClaw/node child processes were much smaller at the time; the growth was in /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled.

Crash screen

The same failure shape also produced this Codex crash screen:

Visible crash details:

Codex crashed with the following error:
(code=null, signal=SIGKILL)
Most recent error is a WARN from a slow SQL log insert:
- message: slow statement execution time exceeded alert threshold
- summary: INSERT INTO logs (...) VALUES ...
- rows_affected: 31
- rows_returned: 0
- elapsed: about 1.343s
- slow_threshold: 1s

After a tool-level aborted read, the app-server PID had changed/restarted and was back around a few hundred MB RSS, which is consistent with the UI crash/reload path rather than a clean handled error.

Thread-management API repro

The Codex thread-management API can list both problematic threads, but reading the pathological one aborts/crashes:

codex_app.list_threads(query="Restart Openclaw") succeeds and returns:
- thread id: 019df821-2a2e-7a01-be45-04b76d73c715
- title: Restart Openclaw
- status: notLoaded
- cwd: /Users/.../LaterAI
codex_app.list_threads(query="OpenClaw recovery root-fix") succeeds and returns:
- thread id: 019ee577-5cd2-75d1-a997-ffb4bf6eeefd
- status: notLoaded
codex_app.read_thread(threadId="019ee577-5cd2-75d1-a997-ffb4bf6eeefd", turnLimit=1, includeOutputs=false) succeeds, although it returns/truncates a very large single-turn summary.
codex_app.read_thread(threadId="019df821-2a2e-7a01-be45-04b76d73c715", turnLimit=1, includeOutputs=false) returned aborted.

This makes the repro more concrete: metadata listing works, a smaller huge recovery thread can be read, but reading the pathological 11.88 GiB Restart Openclaw rollout via the thread-management API aborts/crashes Codex.

Local state correlated with the spike

The local Codex state contains one extreme rollout/thread history outlier:

Rollout JSONL size: 12,754,101,031 bytes (~11.88 GiB)
Line count: 84,643
Thread DB tokens_used for that thread: 1,985,738,719
Another related recovery thread had tokens_used = 40,431,073
~/.codex/sessions total size: ~12 GB, dominated by the single rollout file.
~/.codex/logs_2.sqlite: ~222 MB plus ~63 MB WAL at time of inspection.

Content-safe sampling of the huge JSONL:

Head sample, first 20,000 lines:

response_item: 13,922
event_msg: 5,526
turn_context: 343
session_meta: 175
compacted: 34
token_count events: 2,779
largest sampled line: ~11.1 MB

Tail sample, last 2,000 lines:

response_item: 1,502
event_msg: 467
turn_context: 17
compacted: 14
token_count events: 248
largest sampled line: ~60.9 MB

Suspected code paths

These paths appear capable of loading or scanning large rollout histories without size/record caps.

thread/read with turns loads persisted history and builds API turns

codex-rs/app-server/src/request_processors/thread_processor.rs
thread_read_response_inner -> read_thread_view -> load_persisted_thread_for_read(... include_turns ...)
load_persisted_thread_for_read calls thread_store.read_thread(... include_history: include_turns ...)
When history is returned, it calls build_api_turns_from_rollout_items(&history.items).

Thread store include_history loads the entire rollout into memory

codex-rs/thread-store/src/local/read_thread.rs
attach_history_if_requested calls load_history_items(&path).
load_history_items calls RolloutRecorder::load_rollout_items(path) and stores all items in StoredThreadHistory.

RolloutRecorder::load_rollout_items is unbounded

codex-rs/rollout/src/recorder.rs
It streams lines from open_rollout_line_reader, parses each line, then does items.push(item) into a Vec<RolloutItem> with no file-size cap, record-count cap, byte budget, or cancellation/backpressure noted in the inspected code.
On an 11.88 GiB rollout with some 10-60 MB JSONL records, this can plausibly explain a 30-40 GB app-server footprint.

thread/list normally preserves scan-and-repair behavior unless state DB-only is requested

codex-rs/app-server/src/request_processors/thread_processor.rs accepts use_state_db_only but defaults it through from the client params.
codex-rs/thread-store/src/local/list_threads.rs calls RolloutRecorder::list_threads when use_state_db_only is false, and only calls list_threads_from_state_db when true.
The protocol schema notes that useStateDbOnly skips scanning JSONL rollouts to repair thread metadata, which suggests the default thread list path can still touch rollout files.
The list summary path appears more bounded than full history loading, but it is still part of the startup/listing hazard for pathological rollouts.

Expected behavior

A pathological local rollout should not be able to make the app-server consume tens of GB. Suggested guardrails:

Refuse, truncate, quarantine, or summarize-only any rollout above a sane byte/record threshold for UI list/read/resume operations.
Make thread/list use state DB-only by default in the desktop app when state DB is available, or repair in a bounded background job.
Make thread/read include_turns=true enforce a byte/record budget and return a clear error or partial/summarized history instead of loading the whole rollout into memory.
Add memory/size telemetry around rollout replay/list/repair paths.
Consider clamping or treating impossible token counts (for example ~2B tokens) as corrupt/outlier state.

Reproduction shape

I do not have a minimal synthetic repro yet, but the local state suggests this shape:

Have a very large rollout JSONL under ~/.codex/sessions (observed: 11.88 GiB, 84,643 lines, individual lines up to ~60.9 MB).
Ensure state DB references the thread and contains an extreme token count.
Restart Codex Desktop.
Open the thread list and/or read/resume the pathological thread with turns/history included.
Observe /Applications/Codex.app/Contents/Resources/codex app-server --analytics-default-enabled memory grow to 30-40 GB and swap climb into tens of GB, or observe the app-server abort/crash with SIGKILL.

What would help users immediately

A small safe-mode/quarantine behavior would be very helpful: if Codex detects a rollout above a threshold during startup/list/read/resume, skip full replay, show a recoverable warning, and avoid loading that file into app-server memory.

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

Transport failed: Space limit exceeded: Message too long #29391
Desktop Computer Use thread becomes unrecoverable after appshot/image history poisons remote compaction #28150

Powered by Codex Action

Still reproducible on a newer Codex Desktop release, with an even larger local rollout. This appears to be the same failure mode.

Environment

Codex Desktop: 26.707.62119
Bundled codex-cli: 0.144.2
macOS: 15.7.4 (24G517), arm64

Observed behavior

The issue has recurred for several days.
Fully restarting ChatGPT/Codex only reclaims memory temporarily.
After one restart, the bundled codex app-server launched at 10:05:13 and reached 26-27 GB by about 10:14.
Activity Monitor showed 27.19 GB; sample reported a 32.7 GB peak.
A live footprint capture while memory was high reported about 23 GB physical footprint, including about 21 GB dirty in MALLOC_MEDIUM and 832 MB in MALLOC_LARGE.
The app-server's MCP/Node child processes were only a few MB each, so the growth was in the main bundled codex process.
System pressure was severe: about 12 GB in the compressor and only roughly 167-300 MB unused during captures.

Correlated local state

I checked file metadata only and did not inspect or upload transcript contents.

Largest rollout JSONL: 29,138,057,205 bytes (about 27.1 GiB)
The file was originally created under the 2026/07/05 session directory and was last modified on 2026-07-14 10:01:22 +0900.
Other large rollouts were approximately 2.17 GB, 1.34 GB, and 1.17 GB.

A 3-second live process sample while the problem was occurring showed a Tokio worker in async file reading plus repeated JSON parsing/copy/allocation paths, including:

tokio::fs::file::File as AsyncRead::poll_read
serde_json::read::StrRead::parse_str
_realloc, _platform_memmove, and vm_copy

This strongly matches the unbounded rollout hydration described in this issue. The especially important regression signal is that the problem remains present in Desktop 26.707.62119 / CLI 0.144.2, and restart alone immediately reproduces it because the oversized rollout is loaded again.

No session contents, usernames, project paths, or private logs are included here.

Confirming this on macOS 26.5 (M-series), ChatGPT.app 26.707.72221 / codex-cli 0.144.2 — reproduced on two different MacBook Pros, so it's not machine-specific.

Measurements tonight:

One codex app-server process, 18h uptime → 77 GB footprint (footprint -p), swap 52/53 GB full, macOS 'out of application memory' dialogs.
After force-restarting ChatGPT.app: swap dropped 52 GB → 6.6 GB instantly.
A fresh app-server grew ~860 MB in its first 33 minutes under agent workload (~1.5 GB/hr), so the growth starts immediately, not only after long sessions.
~/.codex/sessions held 1,637 rollout .jsonl files (2.1 GB). Closing sessions in the UI frees nothing — everything lives in the single long-lived app-server heap.

Note for anyone diagnosing via Force Quit dialog: ChatGPT and Google Chrome can show identical multi-GB values (we saw both at exactly 17.98 GB) — that's shared Chromium-framework memory double-counted (the Codex renderer maps ~1.9 TB virtual). Check per-process RSS/footprint before trusting it.

Workarounds that held for us:

Prune old rollouts: move ~/.codex/sessions files older than 14 days elsewhere (2.1 GB → 528 MB here).
A cron/launchd watchdog that kills just the codex app-server process when its footprint exceeds ~12 GB — ChatGPT.app stays open and respawns it on next use, so this is far less disruptive than restarting the whole app.

Would love a real fix: cap/stream rollout history loading and tool output instead of accumulating it in the app-server heap.

Confirming this on a current ChatGPT/Codex Desktop build:

ChatGPT Desktop 26.803.41515 build 6321
bundled codex-cli 0.147.0-alpha.6.5
macOS 26.5.1 (25F80) arm64

The affected process was:

/Applications/ChatGPT.app/Contents/Resources/codex \
  -c features.code_mode_host=true \
  app-server --analytics-default-enabled

macOS recorded a 16 GB peak physical footprint for app-server. At capture time it was still around 5 GB physical with about 4.7 GB swapped out.

vmmap / footprint showed multi-GB allocator regions:

MALLOC_SMALL: 6.5 GB virtual
MALLOC_LARGE: 1.8 GB virtual
main malloc zone allocated: ~3.6 GB
main malloc zone fragmentation: ~1.0 GB

A one-minute trace kept app-server around 98–136% CPU. A 5-second sample repeatedly hit:

tokio::fs::file::File::poll_read
serde_json::read::StrRead::parse_str
_realloc
_platform_memmove
hashbrown allocation/rehash paths

~/.codex/sessions was 7.7 GB. The five largest rollout files were:

2,087,802,775 bytes
1,287,719,134 bytes
1,212,192,775 bytes
1,211,706,260 bytes
1,211,567,011 bytes

lsof showed the 2.09 GB and 1.29 GB rollouts open by the affected app-server during the incident, along with a bunch of smaller rollouts.

Direct helper children were only about 148 MiB RSS combined, so the multi-GB memory use was in app-server itself.

This looks like the same oversized-rollout hydration path described here: app-server is reading rollout history and doing heavy JSON parsing/allocation while multi-GB rollout files are open.

Corroborating report from a current macOS Desktop build. This is the same failure class, with a remote MCP-output-heavy rollout rather than an image-heavy rollout.

Environment

ChatGPT/Codex Desktop: 26.803.61601 (build 6396)
Bundled Codex: 0.147.0-alpha.6.5
macOS: 27.0 (26A5388g), arm64
Hardware: Apple M3 Max, 64 GB RAM
Doctor after recovery: config, state parity, installation, provider/WebSocket reachability, MCP, and update checks all pass

Sanitized evidence

A single active rollout reached 1,697,119,031 bytes with 12,298 valid JSONL records and 0 parse errors.
It was dominated by persisted remote MCP call results: approximately 1.63 GB of the file.
The task had made 2,925 high-volume read calls, including 1,475 calls to one paginated endpoint.
This was not image/base64 history and was not a malformed JSONL case.

Desktop repeatedly resumed the same rollout successfully, then the application crashed about 24 seconds later:

01:04:19Z  thread/resume started
01:04:27Z  resumed with 12,298 items; parse errors: 0
01:04:51Z  ChatGPT/Codex crash

01:06:58Z  thread/resume started
01:07:06Z  resumed with 12,298 items; parse errors: 0
01:07:30Z  ChatGPT/Codex crash

There were three app crashes in approximately four minutes. All three macOS reports show:

Exception: EXC_BREAKPOINT / SIGTRAP
Termination: Trace/BPT trap: 5
Faulting thread: V8Worker

Two earlier codex-code-mode-host reports from the same incident window contain these frames:

v8::base::FatalOOM
v8::Utils::ReportOOMFailure
v8::internal::V8::FatalProcessOutOfMemory
v8::internal::Heap::FatalProcessOutOfMemory

Those code-mode-host failures may be secondary, but they confirm V8 memory exhaustion was occurring during the same high-output task.

Recovery that worked

Backed up and validated the rollout.
Archived the oversized task through the supported Desktop thread archive operation so it was no longer under active sessions.
Archived the other unpinned active rollouts over 1 GB.
Added the supported tool_output_token_limit config setting for future model-visible tool output.
Added workflow limits against full pagination and unbounded sequential high-volume reads.

After the oversized active rollout was quarantined, the app remained stable for more than two hours with no new crash report.

Secondary local-state observation

logs_2.sqlite was 4.42 GB, with about 3.72 GB represented by free pages. Because the database uses incremental auto-vacuum, bounded incremental_vacuum plus WAL checkpoints reduced it to about 0.70 GB while PRAGMA integrity_check remained ok. This was not the primary trigger, but it added avoidable recovery pressure.

Expected guardrails

This incident supports the guardrails proposed here and adds one important case: persisted remote MCP results need byte budgets too, not only shell output, images, or model-visible context.

Suggested fixes:

Enforce byte/record budgets on thread/resume and load history incrementally.
Cap, summarize, or externalize persisted MCP tool results before rollout serialization.
Detect and quarantine one pathological rollout instead of repeatedly crashing the full app.
Expose a supported safe-mode/session-repair path.
Automatically checkpoint and incrementally vacuum the diagnostic log database.

I am intentionally not attaching the raw rollout, full app logs, or unredacted crash reports because they contain private prompts, local paths, and remote API payloads. Three app .ips reports and two code-mode-host .ips reports are available for private upload through the in-product feedback/support path if an OpenAI maintainer wants them.

Same issue here, but at a much lower trigger threshold — this is not just a heavy-user problem.

I hit this on a 16GB M1 Pro during normal daily use (not a power Codex user). A single
106 MB rollout was enough to make the whole system unusable.

Environment

ChatGPT Desktop (bundled Codex): 26.814.41407 (build 6720)
macOS arm64, Apple M1 Pro, 16 GB RAM
Pathological rollout: ~/.codex/sessions/2026/08/03/rollout-2026-08-03T10-56-06-019fc58c-*.jsonl
— 106 MB / 4,613 lines, 6 compactions, 558 tool calls over 15 days
(tiny compared to the 11.88 GiB case, yet triggers the same failure)

Measured impact (12 s sampling while ChatGPT was open)

CPU: 99.6–100% for the entire 12 s window (8 logical cores)
Memory compressor: +229,769 compressions / +58,152 decompressions in 12 s
Swap: grew from ~1.6 GB to ~2.3 GB while ChatGPT was open
Free memory dropped to ~4.4 GB with other apps idle
codex app-server + ChatGPT renderer consumed ~57% CPU

Key point

The same unbounded loading path (RolloutRecorder::load_rollout_items) triggers at
106 MB, not only at 11.88 GiB. On a 16 GB machine already running VS Code/Edge/etc.,
even one long Codex session is enough to stall the whole system. "Archiving" the thread
in the UI does NOT move the file out of ~/.codex/sessions — the app-server still scans
and loads it on startup (verified: file stays in place after Archive).

Request

Please add the guardrails proposed above: byte/record budget for load_rollout_items,
quarantine oversized rollouts, and state-DB-only thread listing. Happy to provide
more data if useful.

I reproduced the same failure class on Linux with Codex CLI 0.151.0, this time after opening three project sessions against one shared app-server.

Sanitized runtime evidence:

the app-server reached about 4.02 GiB RSS and crossed its 4 GiB cgroup high-water mark
the process was heavily memory-throttled/swapping, but there was no OOM kill and this was not the stale-PID/zombie daemon failure mode
five active rollout JSONLs totaled 5.22 GiB; the largest was 2.31 GiB
quarantining only rollouts over 100 MiB and restarting the same direct app-server restored stable WebSocket operation for three simultaneous clients
both the old and rebuilt SQLite state databases passed quick_check; the trigger was oversized full-history hydration, not SQLite corruption

I prepared a minimal fail-closed patch for the full-history path:

adds an opt-in bounded RolloutRecorder loader
rejects legacy full-history hydration over 100 MiB with a clear InvalidRequest
leaves the rollout untouched and preserves metadata-only reads
covers plain JSONL metadata rejection and decoded-size enforcement for compressed rollouts

Draft and commit:

GitHub currently prevents this account from creating a PR directly against openai/codex (CreatePullRequest permission denied), so the clean branch is published in the fork for maintainer review/cherry-pick. Formatting and diff checks pass; the draft records the CI limitation honestly.

I deliberately kept automatic quarantine/migration out of the upstream patch: this change only prevents one pathological thread from taking down the whole app-server and does not modify user history.

Navigation Menu

Repository navigation

Codex app-server can grow to 30-40 GB when local rollout history is huge #29510

Summary

Environment

Observed impact

Crash screen

Thread-management API repro

Local state correlated with the spike

Suspected code paths

Expected behavior

Reproduction shape

What would help users immediately

Environment

Observed behavior

Correlated local state

Environment

Sanitized evidence

Recovery that worked

Secondary local-state observation

Expected guardrails

Environment

Measured impact (12 s sampling while ChatGPT was open)

Key point

Request

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Codex app-server can grow to 30-40 GB when local rollout history is huge #29510

Description

Summary

Environment

Observed impact

Crash screen

Thread-management API repro

Local state correlated with the spike

Suspected code paths

Expected behavior

Reproduction shape

What would help users immediately

Activity

github-actions commented on Jun 23, 2026

pdh0826 commented on Jul 14, 2026

Environment

Observed behavior

Correlated local state

YoungMoneyInvestments commented on Jul 17, 2026

teamleaderleo commented on Aug 10, 2026

adamdmcbride commented on Aug 13, 2026

Environment

Sanitized evidence

Recovery that worked

Secondary local-state observation

Expected guardrails

oneles commented on Aug 19, 2026

Environment

Measured impact (12 s sampling while ChatGPT was open)

Key point

Request

offx366 commented on Aug 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions