macOS: compaction checkpoints re-embed screenshots until rollout hits 3.9 GB; ingestion then uses 10–20 GB RSS, causing jetsam kills and a kernel watchdog panic on a 16 GB Mac #31040

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   appIssues related to the Codex desktop appIssues related to the Codex desktop app

   contextIssues related to context management (including compaction)Issues related to context management (including compaction)

   sessionIssues involving session (thread) management, resuming, forking, naming, archivingIssues involving session (thread) management, resuming, forking, naming, archiving

   browser

   performance

   on Jul 3, 2026
2. github-actions commented on Jul 3, 2026

   

   github-actionsbot

   on Jul 3, 2026 – with GitHub Actions

   Contributor

   More actions

   Potential duplicates detected. Please review them and close your issue if it is a duplicate.

   • VS Code/WSL: repeated compacted snapshots duplicate inline base64 images and cause 1GB+ transcripts / OOM #29812
   • Codex CLI resume can SIGKILL/OOM on huge rollout dominated by repeated compacted records #30932

   Powered by Codex Action
3. chyewon commented on Jul 3, 2026

   

   chyewon

   on Jul 3, 2026

   Last edited by chyewon

   Author

   More actions

   Follow-up: I root-caused this and have a validated patch on a fork branch. Since external PRs are invitation-only per CONTRIBUTING.md, posting the results here — happy to submit it as a PR if the team wants to invite it, or feel free to lift the branch directly.

   Branch: chyewon/codex:fix/rollout-memory-compacted-image-bloat (single commit, codex-rs/rollout only; cargo test -p codex-rollout green, clippy clean)

   Root cause (confirmed in source + live instrumentation)

   1. Disk snowball: compaction persists compacted records whose replacement_history re-embeds all prior inline screenshots (base64 data: URLs inside kept user messages), so every checkpoint re-persists every historical screenshot — 98 checkpoints growing 0.1→53 MB each in my 3.9 GB rollout (mechanism of Desktop compaction repeatedly embeds full image base64 in compacted checkpoints #23257 / Codex session logs grow to 700MB-2GB from repeated compaction history and raw tool output #24948).
   2. Memory: rollout loading is eager (load_rollout_items collects everything into a Vec), and state-DB backfill visits every session file through it (extract_metadata_from_rollout). Measured on a real 973 MB production rollout: backfill peaks at ~1.0–1.6 GB; thread/resume via app-server peaks at 4.8 GB (~5× file size — scaled to the 3.9 GB rollout that predicts ~19.6 GB, matching the jetsam/panic forensics above).

   The patch

   1. Redacts inline data: images >16 KB from persisted compacted checkpoints (schema-valid 1×1 PNG placeholder; in-memory history untouched; small/remote images and non-compacted records unchanged) — stops the snowball at the source.
   2. Streams backfill metadata extraction via a new for_each_rollout_item visitor — backfill memory becomes O(largest line) instead of O(file).
   3. Deserializes rollout lines from a borrow instead of from_value(v.clone()) — removes a per-line full-DOM copy (transient-only win).

   Measured (same 973 MB production rollout)

   Metric	Before	After
   Backfill (state_db::init) peak RSS, same in-process harness	1,604 MB	701 MB (bounded by line size, not file size)
   Rollout size with checkpoint-image redaction	973 MB	130 MB (−87%)
   thread/resume peak RSS on the redacted file	4,825 MB	952 MB (−80%)

   End-to-end checks on the redacted file: thread/resume returns the same thread id with zero parse errors, and a live model turn on the resumed session completed normally (provider accepted placeholder data URLs; the model replied).

   Includes a regression test (recorder_redacts_large_inline_images_in_persisted_compacted_history) covering raw persisted bytes, placeholder round-trip through load_rollout_items, and non-compacted records passing through unchanged.

   ---

   Update (same day): after an adversarial completeness review, the branch gained a second commit (802e92a) extending the persist-time sanitizer beyond message InputImage content to the other image-bearing shapes in replacement_history: FunctionCallOutput content items (FunctionCallOutputContentItem::InputImage) and ImageGenerationCall.result (bare base64), with call_id pairing preserved. The regression test now also asserts the resume entry point (get_rollout_history) consumes a redacted rollout cleanly. cargo test -p codex-rollout green, clippy clean. Known remaining limitation (intentionally out of scope): the resume path still eagerly materializes the full item Vec; with redaction in place files stop reaching GBs, but streaming resume reconstruction would be a worthwhile follow-up.
4. loremcdmx commented on Jul 26, 2026

   

   loremcdmx

   on Jul 26, 2026

   More actions

   Confirming this on macOS, at roughly 10× the scale reported in the opening post. Same mechanism, same log signatures.

   Environment

   • Codex Desktop: ChatGPT.app 26.721.41059, Codex Framework 150.0.7871.128
   • macOS 26 (Darwin 27.0.0), Apple Silicon, 18 cores / 48 GB RAM
   • Observed 2026-07-26

   Scale

   Metric	This issue	Mine
   Largest single rollout	3.9 GB	45.14 GB (48,465,379,354 bytes)
   ~/.codex total	10 GB	463 GB
   ~/.codex/sessions	—	434 GB (433 GB from July alone)
   Rollout files > 100 MB	—	313
   logs_2.sqlite	721 MB	1.5 GB + 192 MB un-checkpointed WAL

   state_5.sqlite was an additional 97 MB with a 4 MB WAL. The disk hit 94% full.

   Evidence that compaction re-embedding is the driver

   After stripping image payloads, the 45 GB rollout is 79,899 lines. The 52,004 base64 PNGs it contained were packed into just 1,018 lines — about 51 images per line. That density is only explicable by snapshots re-embedding the same screenshots, exactly as described here.

   Record breakdown of that file:

   response_item   48,275     compacted       617
   event_msg       28,724     turn_context    764
                              session_meta     52
   

   Head sample of the first 3,000 lines, by payload subtype:

   function_call_output   744 records   34,393,039 bytes   (89% of sampled volume)
   message                 48 records    2,544,401 bytes
   reasoning              452 records      963,614 bytes
   

   Largest single JSON line: 2,274,751 bytes, a function_call_output whose output[0] is {"type":"input_image","image_url":"data:image/png;base64,..."}.

   Across the whole corpus, images were 1.6% of lines but 87.3% of bytes.

   User-visible impact beyond disk

   This is not only a storage problem — it degrades the app:

   [AppServerConnection] response_routed method=thread/turns/list durationMs=40139 queueWaitMs=8735
   [AppServerConnection] response_routed method=thread/turns/list durationMs=56454 queueWaitMs=15548
   

   Opening the affected thread took 40–56 s. The app-server request queue sat pinned at inFlightRequestCount=6 with background requests expiring (reason=queue-expired), and turn/start timed out at 30 s, surfacing as the "Error submitting message" banner (see #27395).

   Result of stripping images

   Replacing every data:image/png;base64,… payload with a 1×1 transparent PNG across 754 rollout files:

   • 534,529 images removed
   • 409.4 GB reclaimed
   • Individual files shrink 97–98% (45.14 GB → 0.63 GB for the worst one)
   • 0 errors; verified line counts unchanged, all lines still valid JSON, and every non-image line byte-identical

   That ratio suggests the actual conversational content of even a very heavy multi-month workload is on the order of a few GB — the rest is duplicated pixels.

   Suggested fix

   Store each image once in a content-addressed blob directory and reference it by hash in the rollout, rather than inlining base64. Failing that, excluding image payloads from compaction snapshots alone would remove the large majority of the growth, since the re-embedding is what turns linear growth into quadratic.

   A checkpoint of logs_2.sqlite/state_5.sqlite WALs on shutdown would also help; mine had 192 MB that never got merged.

   Workaround for anyone hitting this now

   Rewriting each rollout line and substituting the base64 payload with a valid 1×1 transparent PNG keeps the JSON structure intact and the app reads the files normally. Non-image lines can be passed through byte-for-byte. Important: skip any file currently open by the app (lsof -c codex -c ChatGPT -c Codex) — replacing an open file leaves the descriptor pointing at the old inode, so the space is never released.

   Related: #32128, #23257, #25815, #29531, #20864.

   👍React with 👍1Reacted by Glenn 'devalias' Grant