Rollout files store each command's output four times; a 12-day session grew to 1.4 GB #42345

Activity

1. github-actions

   added

   enhancementNew feature or requestNew feature or request

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   sessionIssues involving session (thread) management, resuming, forking, naming, archivingIssues involving session (thread) management, resuming, forking, naming, archiving

   performance

   on Sep 2, 2026
2. github-actions commented on Sep 2, 2026

   

   github-actionsbot

   on Sep 2, 2026 – with GitHub Actions

   Contributor

   More actions

   Potential duplicates detected. Please review them and close your issue if it is a duplicate.

   • Rollouts persist each command's stdout 3x per item_completed record (stdout/aggregated_output/formatted_output) — 60% of session bytes; thread_history stores a 4th copy #41269
   • ~/.codex grew to 42 GB (28.6 GB in a single month): compaction re-embeds the full replacement_history in rollout JSONL, and archived sessions are never cleaned up #41806

   Powered by Codex Action
3. argszero commented on Sep 2, 2026

   

   argszero

   on Sep 2, 2026

   More actions

   Thanks for the precise measurement — this matches the duplication structure I traced in #41269, and your byte counts are the first hard numbers I've seen for the full 4×.

   Where the four copies come from (current main)

   For each user-shell command, execute_user_shell_command (codex-rs/core/src/tasks/user_shell.rs) builds a CommandExecutionItem whose stdout, stderr, aggregated_output, and formatted_output are all cloned from the same execution output. aggregated_output is literally stdout+stderr merged (identical to stdout when there's no stderr — your 24,120/25,280 ratio matches). So one command execution materializes the text up to four times across the rollout:

   1. the custom_tool_call_output-style response_item record (your 289 MB) — model-facing tool result,
   2. CommandExecutionItem.stdout (257 MB),
   3. CommandExecutionItem.aggregated_output (257 MB),
   4. CommandExecutionItem.formatted_output (169 MB — a display-formatted variant of the same text).

   What actually reads each field (checked on current main)

   • aggregated_output — rendered by the resume/TUI exec cell (codex-rs/tui/src/exec_cell/model.rs reads aggregated_output for line counts and transcript rendering). Not droppable without a renderer change.
   • stdout — consumed by the legacy-event conversion path: CommandExecutionItem::as_legacy_request_event (codex-rs/protocol/src/legacy_events.rs) derives ExecCommandEndEvent.stdout from it when serving legacy thread history. Dropping it changes legacy wire output, so it needs the legacy-shape rework first.
   • formatted_output — display-only variant; in the persisted form I could not find any reader (live session display uses the in-memory copy). This is the one copy that is safe to stop persisting today with zero behavioral change.
   • The response_item/custom_tool_call_output copy is a separate model-facing record — it feeds LLM context, so it can't simply go; de-duplicating against it (request 1's "reference it by id") is possible but changes history/wire semantics and needs a design decision from maintainers.

   Already-implemented first step

   I implemented the safe slice on my fork: formatted_output is no longer persisted for new commands (codex-rs/core/src/tasks/user_shell.rs + codex-rs/core/src/tools/events.rs, ~169 MB of your total). Branch fix/rollout-dedup-command-output @ 80d3d79ab7, core tests pass, and it rebases cleanly onto current main. Full details in my #41269 comments. Happy to turn it into a PR if maintainers want that direction.

   Your other three requests

   2. Screenshots as files — reasonable and separate; they're stored inline as base64 text today (your 105 MB). Needs a rollout-schema/migration decision (how old rollouts render), so it's a bigger change than the dedup.
   3. Compaction snapshots — partially tracked in ~/.codex grew to 42 GB (28.6 GB in a single month): compaction re-embeds the full replacement_history in rollout JSONL, and archived sessions are never cleaned up #41806 (re-embedded replacement_history); your 188 MB of 365 × ~527 KB compacted records is the same mechanism.
   4. Size/retention cap — cleanup options exist today (max_rollout_age_days, max_rollouts_per_startup, min_rollout_idle_hours in codex-rs/config/src/types.rs), but there's no per-rollout byte cap or "keep only latest N compaction snapshots" option yet.

   The formatted_output drop is a quick, safe win (~12% of a 1.4 GB file on your numbers); the larger structural dedup (stdout via legacy-event rework, response_item id-referencing, screenshot externalization) each deserve their own scoped change.

   👍React with 👍1Reacted by flip-engineer
4. rknightion commented on Oct 1, 2026

   

   rknightion

   on Oct 1, 2026

   More actions

   Same structure on 0.159.3, so nine versions on from your 0.150.0 this hasn't changed. Two codex homes on one machine, 39.2 GiB of rollout JSONL between them: 7.2 GiB across 1592 sessions and 32 GiB across 7809.

   Byte attribution by field, measured across 60 September sessions totalling 0.34 GiB:

   field	share of all bytes
   item_completed.item.stdout	22.5%
   item_completed.item.aggregated_output	22.5%
   custom_tool_call_output.output	14.8%
   item_completed.item.formatted_output	12.5%

   So 72.3% of the total is shell output. I checked the identity directly across 40 sessions and 3185 items carrying output: 2981 had stdout byte-identical to aggregated_output (93.6%), and 2800 of those matched formatted_output as well (87.9%). Close enough to your 24,120 of 25,280 that I think we're measuring the same thing.

   By record type across 120 September sessions (0.63 GiB), event_msg/item_completed is 69.4% of bytes on its own.

   Your compaction finding holds here too. My largest single file is 346 MB and compacted records are 66% of it across only 19 records, so about 10 MB each.

   For scale, same machine and the same sort of work: codex averages 2.41 to 4.51 MB per session against Claude Code at 0.66 to 0.70 MB. That is 3.4x to 6.8x.

   On your fourth request, there is still no retention setting at 0.159.3. I searched the shipped binary and it carries no retention config key at all. history.persistence and history.max_bytes bound history.jsonl only, which is 764 KB and 4.1 MB in these two homes against 7.2 GiB and 32 GiB in sessions/, so a max_bytes cap misses the growth by four orders of magnitude.

   One thing for anyone writing a cleanup script in the meantime: file mtime is not a usable age signal, because resuming a session refreshes it. Nothing in my tree read as older than 45 days by mtime even though the directories go back to June. The date in the rollout filename is the only one I would trust.

   Requests 1 and 4 are the two that matter from here. Dropping the duplicate copies is the bigger win of the two and it needs no new config surface at all.

   👍React with 👍1Reacted by flip-engineer