Codex session logs grow to 700MB-2GB from repeated compaction history and raw tool output

What version of Codex CLI is running?

0.118.0

What subscription do you have?

Pro

Which model were you using?

gpt-5.5

What platform is your computer?

Darwin 24.6.0 arm64 arm

What terminal emulator and version are you using (if applicable)?

codex-tui

Codex doctor report

What issue are you seeing?

Codex CLI/TUI is creating extremely large session JSONL files under ~/.codex/sessions.

One affected session file is 732 MB:
~/.codex/sessions/2026/03/31/rollout-2026-03-31T22-18-46-019d458c-752d-76c0-bbfa-a29eafa0cbf0.jsonl

A quick breakdown shows the size is dominated by repeated compaction history and raw tool outputs:

337.8 MB from compacted records
230.8 MB from function_call_output
54.8 MB from reasoning
26.4 MB from token_count events
18.0 MB from turn_context

The file has 170,415 JSONL records. The largest single record is about 2.1 MB. My ~/.codex/sessions directory is now about 91 GB total, with 184 JSONL files over 100 MB
and the largest observed file around 1.95 GB.

This looks like unbounded session logging / repeated persistence of compaction history rather than expected transcript storage.

What steps can reproduce the bug?

Use Codex CLI/TUI for a long coding session with multiple tool calls.
Run broad search commands that can produce large outputs, for example rg across large repos or /tmp.
Continue the session long enough for context compaction to occur multiple times.
Inspect ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl.
Check file size and record composition with commands like du -h, wc -l, and jq.

Observed result:
The session JSONL grows to hundreds of MB or more. In one case, a single rollout file reached 732 MB with 170,415 records.

The biggest contributors were:

repeated compacted records containing large replacement_history payloads
raw function_call_output payloads
reasoning and token count event records

What is the expected behavior?

Session logs should stay bounded and should not grow to hundreds of MB or multiple GB for a normal long coding session.

Expected behavior:

Tool outputs should be truncated, summarized, deduplicated, or stored with a size cap.
Context compaction should not repeatedly persist large prior history snapshots in replacement_history.
Old session logs should be compressed or covered by a retention policy.
A long tool-heavy session might produce a larger log, but it should remain within a reasonable size, not 700MB-2GB per JSONL file.

Additional information

Affected file:
~/.codex/sessions/2026/03/31/rollout-2026-03-31T22-18-46-019d458c-752d-76c0-bbfa-a29eafa0cbf0.jsonl

File stats:

Size: 732 MB
Lines: 170,415
Largest single JSONL record: ~2.1 MB
Average record size: ~4.5 KB

Breakdown by record type:

compacted: 337.8 MB across 684 records
response_item/function_call_output: 230.8 MB across 35,368 records
response_item/reasoning: 54.8 MB across 23,659 records
event_msg/token_count: 26.4 MB across 38,885 records
turn_context: 18.0 MB across 955 records
response_item/custom_tool_call: 16.3 MB
response_item/function_call: 14.5 MB

Two large rg tool outputs in the affected session:

~1.9 MB output from an rg command across local repos
~1.3 MB output from an rg command across /tmp and a local repo

Broader local impact:

~/.codex/sessions total size: ~91 GB
JSONL files over 100 MB: 184
Largest observed session JSONL: ~1.95 GB

Session metadata:

originator: codex-tui
affected session cli_version: 0.118.0
current installed CLI: codex-cli 0.134.0
current binary path: /opt/homebrew/bin/codex
affected session records model_provider: openai, but no concrete model ID is present

Please try reproducing with the affected CLI version 0.118.0 by downloading/installing that version, running a long tool-heavy Codex TUI session, and then inspecting the
generated ~/.codex/sessions/.../rollout-*.jsonl file size and record breakdown.

We've decided to eliminate the TUI logs entirely. They were largely redundant because we now log to a sqlite db, and those logs are properly rotated and pruned, whereas the TUI logs were append-only. This change will be in the next release.

i am not sure why this closed while its still an issue. The TUI also causing the issue. my codex sessions are in GB now.

Reopening because I misinterpreted the original bug report. I thought you were referring to logs (as indicated in the title), but upon closer read, you were talking about session rollout files. These rollout files are designed to record the full session details, so this isn't a bug. It's unusual for sessions to be so large that they consume hundreds of MB, but this is certainly possible if you use the same session repeatedly or use /goal to drive the agent to work for long periods of time.

We are looking to add compression to reduce the disk footprint. We're also going to add a delete command if you want to remove specific session rollouts completely.

Adding a cross-link because this issue covers large rollout JSONL growth generally, and the more specific mechanism is tracked in #23257.

We inspected an affected local rollout that reached about 11.36 GiB. The detailed sanitized breakdown is here: #23257 (comment)

Short version:

287 compacted records contributed about 10.68 GiB of the 11.36 GiB rollout.
The compacted records were structured JSON, not opaque blobs: each had payload.replacement_history containing response items.
The largest compacted record was about 52.1 MiB.
Across all compacted records, 43 unique input_image payloads totaling about 51.3 MiB were copied as 9,173 image parts across compaction snapshots.
Those repeated input_image.image_url copies accounted for about 10.66 GiB on disk, roughly ~213x amplification of the unique image payload footprint.

So this is not only "long session produces a large transcript". At least for this sample, append-only rollout persistence plus full replacement_history snapshots turns retained inline image payloads into runaway disk growth.

The latest version of the CLI (0.141.0) includes a codex delete <session ID> command that you can use if you want to delete specific sessions that have grown really large. I realize that's only part of the solution here, but I wanted to mention it in case it's of help to anyone who wants to free up some disk space.

note that you can also just ask codex to salvage these sessions. my 11.36gb session is now 700mb with all the images moved to the last compaction row.

@banteg

@banteg I've done that, but with long-running /goal sessions, the session data keeps accumulating and both the JSONL logs and SQLite db continue to grow over time. I just have a cron job which prunes every few days..

@etraut-openai

Following up with data specific to the /goal angle @etraut-openai flagged back in June ("if you use... /goal to drive the agent to work for long periods of time").

Confirmed reproducible on my end: one /goal thread ran continuously for 14 days without ever rotating, growing to 613,100 lines / 202MB, with 126,748 exec_command calls. Cross-checking ~/.codex/goals_1.sqlite (thread_goals table), that single thread burned 559,871,882 tokens before hitting usage_limited. Checking all my goal threads: 6 of 12 have independently hit usage_limited, totaling 929M of ~1.05B cumulative tokens. This isn't an edge case - it looks like the default outcome for any /goal run left unattended for more than a few days.
One structural detail that might matter for a fix: thread_goals uses thread_id as its primary key, so a goal is permanently 1:1 bound to a single rollout/thread. There's no supported way to continue a goal in a fresh (compact) thread - only start an entirely new goal, losing continuity. If goal-driven work is meant to run long, it probably needs a goal_id-keyed continuation path that survives a thread rotation, not just better compression of the existing one.
In the meantime I built a stopgap using the new PostToolUse/PreCompact/PostCompact hooks -- PostToolUse truncates+archives large exec output before it's ever recorded so it can't be resent on later turns, PreCompact warns inside the session when a goal thread's already deep in goals_1.sqlite token debt...

Reopening because I misinterpreted the original bug report. I thought you were referring to logs (as indicated in the title), but upon closer read, you were talking about session rollout files. These rollout files are designed to record the full session details, so this isn't a bug. It's unusual for sessions to be so large that they consume hundreds of MB, but this is certainly possible if you use the same session repeatedly or use /goal to drive the agent to work for long periods of time.

We are looking to add compression to reduce the disk footprint. We're also going to add a delete command if you want to remove specific session rollouts completely.

Makes sense since I have 14GB of Codex session history since using GPT Pro 20x 5.6 Sol (X-high/Ultra) in Goal Mode; that means in the last 2 days I had 14 GB of memory eaten (yes, this was token-heavy work, I used 4 resets and ~1.5B tokens in one day alone).

and temporarily solved with a symlink to an external drive

Additional sanitized reproduction from macOS 26.4 (25E246), arm64.

Current installed binary is codex-cli 0.144.5. Most of the affected rollout files were authored by codex-tui 0.144.0-alpha.4, with smaller amounts from 0.144.0 through 0.145.0-alpha.16.

Snapshot of ~/.codex/sessions:

total on disk: about 12.3 GiB
377 subagent rollout files account for about 11.9 GiB
297 subagent files authored by 0.144.0-alpha.4 alone account for about 11.7 GiB
non-subagent CLI/Desktop rollout files are only a few hundred MiB combined

A representative subagent rollout is 101,184,191 bytes (96.5 MiB). Serialized-record breakdown:

60 compacted records                    77.3 MiB
12,378 event_msg/token_count records     8.4 MiB
89 event_msg/mcp_tool_call_end records   3.1 MiB
1,121 event_msg/patch_apply_end records  2.9 MiB
138 response_item/message records         1.9 MiB

Its session metadata identifies the source as source.subagent.thread_spawn. This suggests an additional amplification path beyond a single long-running root thread: subagent rollouts inherit substantial parent context, then repeatedly persist that context in compacted.payload.replacement_history. In this sample, compacted records alone are roughly 80% of the file.

No inline images dominate this particular sample. The dominant data is repeated compaction history. This remains observable in rollout files from the 0.144.x line, not only the 0.118.0 version in the original report.

I investigated this on a real, privacy-sensitive workload and have a small, tested compression patch ready. I am posting the analysis first, per the invitation-only contribution policy, and would appreciate maintainer alignment/invitation before opening a PR.

Impact

On Codex 0.144.2, ~/.codex/sessions reached 75,412,910,173 bytes (70.23 GiB) across 644 rollouts after roughly one to two months of heavy use:

65.07 GiB was in subagent rollouts.
One parent thread had 365 child rollouts totaling 62.62 GiB; the parent itself was 1.22 GiB.
July 17 alone accounted for 29.15 GiB, and July 18 already accounted for 27.42 GiB at the time of the audit.

The audit is content-private: it reports sizes, JSON paths, counts, and hashes, and never prints message/tool contents.

What is being duplicated

I recursively analyzed a representative 760,247,640-byte (725 MiB) child rollout:

compacted records: 652.06 MiB / 89.936% of the file.
data: URLs: 659.78 MiB.
payload.replacement_history[].content[].image_url: 632.39 MiB.
repeated strings of at least 256 bytes within the file: 651.49 MiB.
one 977,978-byte image string occurred 138 times (about 134 MiB attributable to that value alone).

Exact-line deduplication would save only about 1.10 MiB because timestamps and surrounding metadata differ. This is content/history-level duplication, consistent with #23257.

There is also substantial parent/child duplication. Comparing strings of at least 256 bytes by SHA-256, the child contained 705.39 MiB of large strings and 704.50 MiB already existed in its parent: 99.874% overlap by bytes (98.434% by unique values).

Source trace

At current main (56395bddaf26eb2829387ca6a417bf9128e5b239 when tested):

session::replace_compacted_history persists replacement_history: Some(items.clone()); local/remote/v2 compaction paths all create full replacement histories.
paginated subagents call LiveThread::create_with_inherited_model_context; SessionMeta.history_base exists, but the observed legacy rollouts still physically contain inherited context and the common creation/extraction paths leave it unset.
transparent .jsonl.zst reading/materialization and a cold-compression worker already exist in rollout/src/compression.rs, but LocalThreadStoreCompression is under-development/default-off, waits seven days, and uses zstd level 3 with its default match window.

The current compression infrastructure is a good base, but default zstd cannot see repeated blocks separated by tens/hundreds of MiB. Also, a fixed seven-day delay does not help a workload that generated more than 50 GiB in two days.

Real-file compression benchmark

All rows use the same 760,247,640-byte rollout; full round trips were byte-identical by SHA-256.

Encoding	Compressed bytes	Ratio	Representative resource result
gzip -6	498.3 MB	1.53x	20.62 s
zstd -3 default	463,794,252	1.64x	0.43 s, about 101.8 MiB RSS
zstd -3, long=23	108,474,356	7.01x
zstd -3, long=24	72,554,649	10.48x
zstd -3, long=25	52,901,838	14.37x	compress about 0.32 s / 58.3 MiB RSS; decompress about 0.20 s / 37.9 MiB RSS
zstd -3, long=27	32,856,569	23.14x	compress about 0.34 s / 166 MiB RSS; decompress about 0.19 s / 138.8 MiB RSS

I selected a 32 MiB maximum window (windowLog 25) rather than the best ratio. It gives most of the storage benefit while keeping encoder/decoder memory far below long=27 and is appropriate for two background jobs.

Focused patch ready

The local topic branch changes only codex-rollout (55 inserted lines across two files):

set pledged source size and include a frame checksum;
for rollouts >= 8 MiB, enable long-distance matching;
choose windowLog = ceil(log2(source_size)), clamped to 23..25;
add a deterministic worker test using one incompressible 4 MiB block repeated three times; assert the result is less than half the source and decodes exactly.

Validation:

just test -p codex-rollout: 100/100 passed;
just fix -p codex-rollout: passed;
just fmt: passed;
git diff --check: clean.

I also verified the installed 0.144.2 app-server can thread/read both a synthetic and a real compressed rollout without materializing it. A guarded local migration compressed 194 closed rollouts, verified every frame and decompressed SHA-256, skipped six files observed open, and explicitly saved 53.78 GiB. The session directory fell from 70.23 GiB to about 19.3 GiB while preserving logical JSONL bytes. These were cold candidates and no integrity failures were observed, but lsof double checks cannot theoretically close an open-after-check race. A general migration should stop Codex writers, and the automatic worker should resolve the representation/append coordination called out in #25089 before default enablement.

Release compatibility follow-up

I downloaded official macOS arm64 release binaries, verified each asset against the SHA-256 digest in GitHub Release metadata, and ran the same isolated app-server probe through both thread/read and thread/resume:

Stable release	Read `.jsonl.zst`	Resume/materialize plain JSONL
0.135.0	No; treats the frame as ordinary UTF-8 JSONL	No
0.136.0	No; treats the frame as ordinary UTF-8 JSONL	No
0.137.0	Yes	Yes
0.141.0	Yes	Yes
0.144.2	Yes	Yes
0.144.6	Yes	Yes

So 0.137.0 is the earliest tested downgrade-compatible stable release; merge/release timing alone would have incorrectly suggested 0.136.0. The local migration tool now requires an actual supplied Codex binary and refuses to modify storage unless its isolated read-and-resume probe passes.

Proposed sequencing

Land the bounded long-distance zstd change as a schema-neutral, independently reversible improvement.
Resolve writer/compressor coordination, then separately decide when to stabilize/default-enable local compression and whether large closed/subagent rollouts should use a size-aware cooldown rather than seven days.
Design the structural fix: immutable parent-history prefix references and/or a versioned content-addressed blob/chunk store for large payloads. That needs transactional writes, dual-read compatibility, reference-aware deletion, and generation/lease-based GC; I would not mix it into the first patch.

Would this focused compression change align with the maintainers' intended direction? If so, I would be happy to submit and own the PR; please explicitly invite it per docs/contributing.md. I can also provide the privacy-preserving audit tool and more benchmark details if useful.

Hit this too. My sessions were hitting 2GB and looping forever reporting fake success. I'm building a tiny open source watchdog that auto kills loops, rotates logs, and verifies real success. Want me to tag you when I push the fix tomorrow?

Independent corroboration from a recovered/quarantined WSL profile. I am not attaching rollout contents because they contain private development conversations, but I parsed record types and byte counts without emitting payload values.

Environment/context:

Ubuntu 24.04 under WSL 2
Affected history produced by Codex CLI 0.144.x (one incident session metadata records 0.144.3; the later installed/resumed CLI was 0.144.6)
483 rollout JSONL files
Total sessions/ size: 34,934,047,187 bytes (34.9 GB decimal)

Largest rollout:

2,363,926,002 bytes
89,511 JSONL lines
0 invalid JSON lines; file had a complete newline terminator
compacted: 212 records / 2,123,731,427 bytes (~89.8% of the file)
response_item/function_call_output: 17,545 records / 155,589,120 bytes
event_msg: 27,910 records / 25,611,715 bytes
remaining record classes were much smaller

A separate active rollout involved in the recovery incident was 573,845,348 bytes with 61,406 valid JSONL lines and no malformed tail. The quarantined Codex SQLite databases also passed full PRAGMA integrity_check.

This strongly matches the issue's repeated-compaction-history mechanism rather than ordinary transcript size or malformed writes. I found no recovered config override pointing to intentionally enabled trace/rollout verbosity.

Important separation from the incident that led to this forensic review: the WSL distro's host VHDX later suffered an independently evidenced NTFS compressed-file fragmentation/extent-limit failure. That host storage failure is being reported separately to Microsoft. It did not create the 2.36 GB valid JSONL structure described above; the bloat was already valid, structured Codex persistence. Conversely, I am not claiming Codex caused the VHDX failure.

Suggested acceptance criteria remain:

Do not repeatedly embed full replacement/compacted history in every checkpoint.
Bound or rotate active rollout files before they reach hundreds of MB/GB.
Make resume/listing incremental and tolerant of oversized histories.
Surface local-state growth in diagnostics/UI before it consumes tens of GB.
Handle persistence I/O failure explicitly so one filesystem error cannot silently strand a session.

I can provide additional aggregate-only breakdowns if maintainers need them, but cannot post private rollout payloads publicly.

I’m seeing the same unbounded rollout growth, with a particularly severe subagent fan-out amplification pattern in Codex Desktop.

Environment:

macOS arm64
affected sessions: Codex Desktop / codex-cli 0.145.0-alpha.27
current Desktop session runtime: 0.145.0-alpha.30
current standalone CLI: 0.144.6

Local structural findings (no private rollout data attached):

~/.codex/sessions: 135 GB total
only 138 JSONL files
96 GB created on one day and 34 GB the preceding day
57 files over 1 GB, with dozens around 2.5 GB each
one representative 2.5 GB rollout had 36,830 lines and 190 type:"compacted" records
largest individual compacted JSONL records were about 14.9 MB
latest inspected compacted replacement_history contained 22 distinct user clipboard PNGs as inline data:image/png;base64,... input_image items
those 22 clipboard image names occurred 10,770 times in that single rollout due to repeated snapshots

The additional multiplier is subagent inheritance:

56 of the oversized files were Desktop thread_spawn subagent sessions created from the same image-heavy parent
each child inherited the parent’s inline image history and then repeatedly persisted it again during compaction

This appears closely related to #24550 and #24388, but shows that an image-heavy parent plus subagent fan-out can turn a small number of pasted screenshots into more than 100 GB of rollout data. It also produced substantial disk/CPU activity consistent with #20864 because Desktop scans the hot sessions tree.

Expected behavior:

Compacted history should replace historical inline image data with bounded markers/references.
Spawned subagents should not clone raw base64 image history into every child rollout.
Rollout persistence should deduplicate blobs and enforce per-session/total retention limits.
Desktop should avoid eagerly parsing all large rollout files.

I can provide additional redacted structural statistics if useful, but the raw files contain private prompts, local paths, and screenshots.

Windows Codex Desktop corroboration, with an explicit limitation: during the #35032 compaction/reprocessing incident, the workstation was reported near capacity while the affected long-running thread accumulated repeated compaction history, raw tool output, screenshots, and evidence artifacts. Exact .codex/sessions byte attribution has not yet been measured, so this is not presented as a quantified reproduction of the 700MB–2GB figures here. The visible behavior does match the same retention pattern: several Context automatically compacted events, immediate ~80% post-compaction occupancy, and repeated historical/tool state being carried forward. Please consider Codex Desktop/Windows session retention when fixing this report, not only CLI/TUI.

Current-version Desktop data point, captured 2026-07-25 with privacy-safe record-type aggregation only.

Environment: macOS arm64, ChatGPT/Codex Desktop 26.721.41059, bundled codex-cli 0.146.0-alpha.3.1, history_mode: legacy.

One actively written 111,879,772-byte rollout on the current runtime was dominated by:

response_item/function_call_output: 80.6 MiB across 440 records
compacted: 9.6 MiB across 7 records
event_msg/mcp_tool_call_end: 6.7 MiB across 96 records
response_item/message: 2.3 MiB across 136 records

Separately, a historical 598 MB depth-1 child was dominated by inline image-generation results and custom tool outputs. The full session store is currently 122.2 GiB.

This indicates raw function/tool-output persistence remains a material growth path on 0.146.0-alpha.3.1, even when compaction is not the largest category in the active rollout. Per-record caps plus artifact references/deduplication would address both this path and the image-heavy cases.

While we're waiting for this to be resolved... has a kind soul perhaps written (with Codex hehe), a JSONL cleaner-upper? Something that blasts through all the JSONLs and surgically removes certifiable junk? I would do it but I don't know what separates junk from JSONL lines that are needed to keep my session integrity. There has to be a way to throw away 95% of this. If using all this space were a necessary thing, Claude would do the same, but it doesn't.

I can reproduce the same unbounded session-log growth in the Codex Desktop app on macOS, not only in CLI/TUI.

Environment

ChatGPT/Codex desktop app: 26.721.41059
Bundled Codex CLI: 0.146.0-alpha.3.1
macOS on Apple Silicon, 18 GB RAM

Observed impact

A child design-auditor subagent inside an active parent task wrote one live .jsonl transcript of 2,787,011,429 bytes (~2.6 GiB).
The live ~/.codex/sessions store reached ~3.2 GiB.
Codex then repeatedly launched concurrent raw-history searches over ~/.codex/sessions; I observed five rg -l --no-ignore --glob '*.jsonl' searches at once.
On an 18 GB Mac this produced severe responsiveness problems: multi-GB swap use, several GB of memory compression, high system load, and desktop freezing.

The key point is that the parent task was still actively working. The oversized file was a child-agent transcript, so the user cannot safely treat it as disposable or archive it just because the child pass emitted task_complete.

No raw session log is attached because it contains private project material. I can provide redacted record counts or diagnostics on request.

Please add app-side protections: bound/rotate session persistence, cap or deduplicate raw tool output and repeated compaction snapshots, avoid full raw-history scans in the desktop app, and provide safe user-visible storage controls.

Here's a totally vibe-coded Bash script for Mac OS to solve the problem, without losing your chat history or tripping up Codex UI (mysteriously lost chats). I found that the .JSONL files contain copies of copies of copies of any and all PNGs (proof screenshots), some PNG-containing JSON lines are up to 10 MB big! They multiply like rabbits.

Rather than questioning why, I decided to replace them with a 8 x 8 pixel placeholder PNG of a small red ❌ right in the JSONL text. It seems to have no negative effect; I can still load all PNGs embedded inline in the chat history, but even if some got replaced with a placeholder, these proof screenshots that Codex generates have little future value and storing them as BASE64 is insane anyway.

Step 1: Before you trust anyone's script, even mine, review it before you run it!
Step 2: Throw this script somewhere where your Mac OS Terminal can find it (some place that's in $PATH) and set execute permissions (chmod +x shrink-codex-jsonl-dir.sh)
Step 3: cd to one of your date folders that has grown excessively large, e.g. cd ~/.codex/sessions/2026/06/27
Step 4: Execute the script, it will only process .jsonl files >= 1 GiB and will list affected files and pause before processing each file:

It loops through the candidate files, telling you in advance what it will do, and asking you to press any key before acting.
It then replaces all BASE64 PNGs in the JSONL file with the little ❌-looking one, up to today's date minus 7 days. Younger PNGs are ignored so worst case you still have your last week's worth of PNGs.
It renames, compresses & verifies the old file with rar (brew install --cask rar) and deletes the old file after compression was verified. So you still keep your original file, should something go terribly wrong. The slimmer file replaces the old file. The old/big JSONL files compress really well so you might as well keep the .rar file, or if space is really tight, move it to some external disk.
It then uses applesauce (brew install applesauce) which uses the Mac OS APFS or HFS+ compression to compress the newly-reduced file transparently on disk for an extra compression of about 50%. This is the little-known equivalent of NTFS file compression, just on Mac OS. The problem is, it can't compress files over 4 GiB and many of my Codex JSONLs exceed 5 GiB. After such files have been stripped of their BASE64-encoded PNGs, they are well under 4 GiB, so applesauce can do its work.

That's it, once you poke around in your sessions folder you will see that very few "day folders" have the biggest files (often over 5 GiB) and some folders have (almost) nothing. So while a bit tedious, in the name of causing minimum damage, focus on one month (like June), and find the biggest day folders there and run this in them.

No warranty of any kind — if this deletes your home folder or sends your crypto keys to North Korea, not my responsibility!

#!/usr/bin/env bash
#
# shrink-codex-jsonl-dir.sh — process a directory of JSONL rollout logs.
#
# For each FILE.jsonl (excluding *-with-pngs.jsonl):
#   1. rename  FILE.jsonl            -> FILE-with-pngs.jsonl
#   2. build   FILE.jsonl            (same content, PNG payloads replaced by a tiny X)
#   3. rar     FILE-with-pngs.jsonl  -> FILE-with-pngs.jsonl.rar   (verified, source removed)
#   4. applesauce compress /absolute/path/FILE.jsonl
#
# Usage:
#   ./batch-shrink-rollouts.sh [DIR] [DAYS_TO_SPARE] [MIN_PAYLOAD_LEN]
#
#   DIR              directory to scan (default: ./)
#   DAYS_TO_SPARE    records newer than this keep their images (default: 7; 0 = shrink all)
#   MIN_PAYLOAD_LEN  only replace payloads at least this long (default: 500)
#
# Only files >= MIN_SIZE (default 1 GiB) are processed; smaller ones are listed
# and skipped, since the rename/rar/applesauce cycle is not worth it for them.
#
# Environment:
#   YES=1            skip the per-file keypress prompt
#   DRY_RUN=1        list what would happen, change nothing
#   EXT=jsonl        file extension to match (default: jsonl)
#   MIN_SIZE=...     minimum file size in bytes to bother processing
#                    (default: 1073741824 = 1 GiB)
#   NO_EARLY_EXIT=1  do not stop scanning at the first recent record
#
set -uo pipefail

DIR="${1:-./}"
DAYS="${2:-7}"
MIN_LEN="${3:-500}"
EXT="${EXT:-jsonl}"
MIN_SIZE="${MIN_SIZE:-1073741824}"          # 1 GiB
ASSUME_YES="${YES:-0}"
DRY="${DRY_RUN:-0}"
EARLY=$([ "${NO_EARLY_EXIT:-0}" = "1" ] && echo 0 || echo 1)

for tool in gawk rar; do
  command -v "$tool" >/dev/null 2>&1 || {
    echo "error: $tool not found (brew install $tool)" >&2; exit 1; }
done
command -v applesauce >/dev/null 2>&1 || echo "warning: applesauce not found — step 4 will be skipped" >&2

[ -d "$DIR" ] || { echo "error: '$DIR' is not a directory" >&2; exit 1; }
DIR_ABS=$(cd "$DIR" && pwd -P) || exit 1

CUTOFF=$(date -u -v-"${DAYS}"d +%Y-%m-%dT%H:%M:%S.000Z 2>/dev/null) \
  || CUTOFF=$(date -u -d "${DAYS} days ago" +%Y-%m-%dT%H:%M:%S.000Z 2>/dev/null)
[ -z "$CUTOFF" ] && { echo "error: could not compute cutoff date" >&2; exit 1; }

XPNG='iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIBAMAAAA2IaO4AAAAHlBMVEXtPj4AAAD+R0f6amr1EBDtPj7yQEDuQEAAAAAAAABBQS5NAAAACHRSTlOTAP8RCmx4kng7J/sAAAAxSURBVHjaY3Q9L2vEwvuW+T0zO8fzz0yCj+QEmD4wsP9n1vh//yeT0ENRJWbenzJWABOYDeld4+bdAAAAAElFTkSuQmCC'

fsize() {    # portable size in bytes: BSD stat, then GNU stat
  stat -f%z "$1" 2>/dev/null || stat -c%s "$1" 2>/dev/null
}

human() {    # bytes -> human, no external deps
  LC_ALL=C awk -v b="$1" 'BEGIN{
    split("B KiB MiB GiB TiB", u, " "); i=1
    while (b >= 1024 && i < 5) { b /= 1024; i++ }
    printf (i==1 ? "%d %s" : "%.2f %s"), b, u[i]
  }'
}

shrink() {   # shrink <input> ; writes to stdout
  LC_ALL=C gawk -v x="$XPNG" -v minlen="$MIN_LEN" -v cutoff="$CUTOFF" -v early="$EARLY" '
  {
    ts = substr($0, 15, 24)
    if (ts !~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}T/) {
      head = substr($0, 1, 200)
      if (match(head, /"timestamp"[ \t]*:[ \t]*"[^"]{10,40}"/)) {
        ts = substr(head, RSTART, RLENGTH)
        sub(/^"timestamp"[ \t]*:[ \t]*"/, "", ts); sub(/"$/, "", ts)
      } else ts = last_ts
    }
    if (ts != "") last_ts = ts

    if (ts != "" && ts >= cutoff) {
      print; while ((getline line) > 0) print line
      if (early) exit
      next
    }
    if (index($0, "png;base64,") == 0) { print; next }

    out = ""; rest = $0
    while (match(rest, /data:image\\?\/png;base64,[A-Za-z0-9+\/=\\]*/)) {
      pre  = substr(rest, 1, RSTART - 1)
      hit  = substr(rest, RSTART, RLENGTH)
      rest = substr(rest, RSTART + RLENGTH)
      c = index(hit, ","); plen = length(hit) - c
      if (plen >= minlen) { out = out pre substr(hit, 1, c) x; n++ }
      else                { out = out pre hit }
    }
    print out rest
  }
  END { printf("       replaced %d payload(s)\n", n+0) > "/dev/stderr" }
  ' < "$1"
}

# ---- collect targets -------------------------------------------------------
targets=(); toosmall=()
while IFS= read -r f; do
  sz=$(fsize "$f"); [ -z "$sz" ] && continue
  if [ "$sz" -ge "$MIN_SIZE" ]; then targets+=("$f"); else toosmall+=("$f|$sz"); fi
done < <(
  find "$DIR_ABS" -maxdepth 1 -type f -name "*.${EXT}" ! -name "*-with-pngs.${EXT}" | sort
)

echo "Directory : $DIR_ABS"
echo "Cutoff    : $CUTOFF  (records newer than this keep their images)"
echo "Min size  : $(human "$MIN_SIZE")"
echo

if [ ${#toosmall[@]} -gt 0 ]; then
  echo "Below size threshold — skipping (${#toosmall[@]}):"
  for e in "${toosmall[@]}"; do
    printf '  %s  (%s)\n' "${e%|*}" "$(human "${e##*|}")"
  done
  echo
fi

if [ ${#targets[@]} -eq 0 ]; then
  echo "No *.${EXT} files at or above $(human "$MIN_SIZE") in $DIR_ABS."
  exit 0
fi

echo "Files to process (${#targets[@]}):"
for f in "${targets[@]}"; do
  printf '  %s  (%s)\n' "$f" "$(human "$(fsize "$f")")"
done
echo

[ "$DRY" = "1" ] && { echo "DRY_RUN=1 — nothing was changed."; exit 0; }

# ---- process ---------------------------------------------------------------
ok=0; failed=0
for src in "${targets[@]}"; do
  base="${src%.$EXT}"
  orig="${base}-with-pngs.${EXT}"
  rarfile="${orig}.rar"

  echo "── $(basename "$src")"
  echo "   1. rename to       $(basename "$orig")"
  echo "   2. write shrunk    $(basename "$src")"
  echo "   3. rar + verify    $(basename "$rarfile")   (original removed on success)"
  echo "   4. applesauce      $src"

  if [ -e "$orig" ] || [ -e "$rarfile" ]; then
    echo "   SKIP: $(basename "$orig") or its .rar already exists"; echo; continue
  fi

  if [ "$ASSUME_YES" != "1" ]; then
    if [ -t 0 ]; then
      printf '   Press any key to continue, or Ctrl-C to abort... '
      read -r -n 1 -s; echo
    else
      echo "   (not a tty — set YES=1 to run unattended)"; exit 1
    fi
  fi

  mv -n "$src" "$orig" || { echo "   FAIL: rename"; failed=$((failed+1)); echo; continue; }

  if ! shrink "$orig" > "$src"; then
    echo "   FAIL: shrink — restoring original"
    rm -f "$src"; mv -n "$orig" "$src"
    failed=$((failed+1)); echo; continue
  fi
  if [ ! -s "$src" ]; then
    echo "   FAIL: shrunk file is empty — restoring original"
    rm -f "$src"; mv -n "$orig" "$src"
    failed=$((failed+1)); echo; continue
  fi
  printf '       %s -> %s\n' "$(human "$(fsize "$orig")")" "$(human "$(fsize "$src")")"

  if ! rar a -t -df -m5 -y -- "$rarfile" "$orig"; then
    echo "   FAIL: rar — shrunk file kept, original still at $(basename "$orig")"
    failed=$((failed+1)); echo; continue
  fi

  if command -v applesauce >/dev/null 2>&1; then
    applesauce compress "$src" || echo "   WARN: applesauce returned non-zero"
  fi

  ok=$((ok+1)); echo "   done"; echo
done

echo "Processed $ok file(s), $failed failure(s)."

Adding an independent data point that matches this diagnosis exactly (codex-cli 0.146.0, macOS arm64).

We dissected a 335 MB rollout accumulated by a single session in 7 days (20,584 lines), accounting bytes per record type:

share	record type
62% (197 MB)	`compacted` — full `replacement_history` snapshots
24% (77 MB)	`function_call_output`
5% (18 MB)	`custom_tool_call_output`
~2%	`reasoning`
rest	messages / token_count / patch events

The 62% is ~90 giant rows (>400 KB each, mostly ~4.6 MB): every auto-compaction appends another full copy of the replaced history. The growth loop we observed: output-heavy session → frequent auto-compaction → +4.6 MB snapshot per compaction → file bloat. So sessions with heavy tool output bloat fastest precisely because they compact most often.

Memory-side impact (the #29510 class): a different session whose rollout had been resume-chained since early May reached 16.2 GB on disk; resuming it ballooned the codex process to ~17 GB RSS and pushed a 16 GB machine deep into swap. Starting a fresh session (breaking the resume chain) dropped the process to ~90 MB.

One observation toward a cheap mitigation: since each compacted row's replacement_history supersedes all earlier history, only the last compacted row plus subsequent records appears to be needed for resume. An offline "vacuum" that drops superseded snapshots would have shrunk our 335 MB file by ~60% and the 16 GB one by roughly an order of magnitude — happy to test a patch or provide more data.

Adding a severe independent reproduction from codex-cli 0.146.0 on Linux x86_64, using Codex Desktop with a remote Linux project.

A single long-lived parent-thread rollout reached exactly 59,589,245,094 bytes (59.6 GB decimal / ~55.5 GiB). The user-visible conversation itself was nowhere near that size.

Failure pattern

After the last normal user/assistant exchange, the thread entered a context-compaction loop and appended approximately 788 further compaction cycles. Representative cycles contained:

one compacted JSONL record around 72.75 MB (observed sizes clustered around 72,747,402–72,756,021 bytes)
world_state (~32 KB)
turn_context (~4.5 KB)
token_count / context_compacted events
a small injected message and reasoning record
then another full compacted record roughly 20–30 seconds later

The tail reported:

total_tokens: 263,976
model_context_window: 258,400

This may be relevant to the trigger: compaction apparently failed to get the thread below the effective context limit, but the agent kept retrying and persisted another ~72.75 MB replacement history each time.

Impact

The thread stopped producing a final answer.
Codex Desktop could no longer open it.
The normal thread-history reader also stalled on the file.
Other small threads continued to work.
Recovery required reverse byte-range scanning of the JSONL, extracting the last coherent messages/assets, and creating a clean continuation thread.
The original oversized thread is being removed after this sanitized report because retaining it wastes ~56 GiB and it contains private conversation/project data.

This also reproduces the unrecoverable-thread impact described in #25215, but with the storage amplification captured directly: repeated giant compacted records turned one thread into a 59.6 GB file.

Expected safeguards

A failed compaction must not retry indefinitely while appending another full replacement history every cycle.
Successfully superseded compacted snapshots should be replaceable/deduplicated or vacuumable.
Add a per-thread size/growth-rate circuit breaker and surface a recoverable error before disk usage becomes catastrophic.
Provide a built-in “continue from latest coherent checkpoint” / vacuum operation.
Thread listing and recent-turn reads should not require hydrating the entire rollout.

I cannot attach the raw JSONL because it contains private conversation history, tool inputs/outputs, and filesystem paths, but the byte counts and record-shape diagnostics above were collected without publishing payload contents.

Adding an independent Windows Codex Desktop data point.

Environment: Windows 11 x64, Codex Desktop (Microsoft Store build 26.803.5235.0).

Observed locally:

The C: drive reached 0 bytes free.
The local Codex directory (C:\Users\<redacted>\.codex) reached approximately 398.8 GB.
sessions accounted for approximately 396.5 GB.
There were 554 rollout-*.jsonl files; the largest was 9.37 GB, with many others in the 2–5 GB range.
A bounded inspection confirmed the rollouts contain session_meta, event_msg, turn_context, response_item, reasoning, function calls/outputs, and tool/web events.
I deleted 273 older rollout files, reclaiming approximately 104.7 GB. Because Codex was still open, approximately 11.6 GB was written during the cleanup; the remaining session tree was still about 293.2 GB across 281 files.

I did not publish any raw rollout content because it contains private prompts, tool I/O, and local project paths.

This appears consistent with the reports here: long-lived/resumed threads and repeated compaction or tool-output persistence can create multi-GB append-only rollouts. The continued multi-GB writes while the desktop app was open are especially concerning.

A per-session size/growth guard, compaction/vacuum of superseded history, and the documented desktop equivalent of codex delete <session ID> would be very helpful for Windows users.

Adding an independent Linux data point that isolates a structural change in 0.147.0 which makes this worse: with the new paginated thread history, the full stdout of every command execution is now persisted, untruncated, as item_completed event records.

Environment: codex-cli 0.147.0, three Linux hosts (Ubuntu server + WSL2), heavy multi-session use.

Observations from one representative rollout (~/.codex/sessions/2026/08/08/rollout-…jsonl):

File total: 87,457,678 bytes.
60.7 MB (69%) are event_msg records, of which 59.6 MB are payload.type: item_completed (2,840 records).
These are CommandExecution turn items whose stdout field embeds the complete aggregated command output (full pytest runs, full rg output). Across our hosts we have seen single item_completed records up to ~2.7 MB.
response_item records add another 23.4 MB, so exec output is effectively persisted twice: once as function_call_output, once inside the completed item.

Mechanism (from source at rust-v0.147.0): should_persist_event_msg in codex-rs/rollout/src/policy.rs persists every EventMsg::ItemCompleted when the thread's history mode is Paginated, and paginated thread history is what 0.147.0 rolled out (#36948/#36950, legacy migration in #37175). Rollouts written by 0.146.0 on the same hosts contain no item_completed records at all; there, ExecCommandEnd is classified as transient and never persisted.

Impact beyond disk usage: anything downstream that consumes rollout JSONLs (we ingest them into an analytics/review pipeline) suddenly receives megabyte-sized event_msg records where event messages used to be small envelopes. Across our three hosts the item_completed payload volume is on the order of 460 MB/day since the 0.146→0.147 upgrade, vs ~10 MB/day of event payloads before.

Bounded persistence for CommandExecution.stdout in persisted item_completed records (head/tail excerpt + byte count, like the model-facing truncation) would address both the disk growth reported here and the downstream breakage.

Additional Windows Codex Desktop evidence:

One archived task has grown to 607.2 MB across 49,947 records.
102 repeated context snapshots consume 454.4 MB; individual snapshots reach 8.5–9.7 MB.
Tool-result records consume another 73.9 MB across two matching record types, indicating duplicate persistence.
logs_2.sqlite reached 265.8 MB and 112,695 rows in approximately ten days.
TRACE and DEBUG logging account for approximately 156 MB, despite no user-configured debug logging.
Archiving the task preserved its history but did not reduce its size.
The Windows app intermittently becomes extremely sluggish while scrolling and typing, even when CPU use is low.

Requested fixes:

Bound and deduplicate repeated context snapshots.
Persist each tool result once and reference it elsewhere.
Add automatic log retention and rotation.
Lazy-load archived tasks instead of processing their complete history.
Provide a supported cleanup or repair function that preserves task history.

Related reports: #21007, #22088, and #34268.

No raw task files or logs are attached because they contain private conversations and project information.

Detect Codex turn ends from its own task_complete event, not a heuristic

Disclosure: This comment was generated by Codex, operating fully autonomously in the research and drafting of this comment. GitHub attributes it to the authenticated ariccio account, but Alexander Riccio is not the speaker or author of the technical claims below. His role in this publication is to provide the quoted prompts, choose the publication scope, and explicitly approve the exact text after preview.

Verbatim user prompts and publication context

Prompt that initiated this issue census and comment drafting:

Locate all the github issues in the repo that are related to #42039, or otherwise may be relevant to #42039, and draft comments that add links to the PR.

Additional public-safe context from the work that prompted this review:

It is blocking major development work on my machine, especially in the important long running worktree, because I don't have enough disk space!

We are struggling too much to recover space and unblock development

Publication context: These excerpts came from a separate attempt to recover disk space safely while preserving valuable Codex history. Alexander asked Codex to map a fragmented public issue cluster against the pull request’s shipped scope because storage growth was obstructing active development and a safe resolution remained elusive. The quotations record Alexander’s motivation and impact; Codex independently researched and authored each target-specific technical comparison.

Publication control: this exact body was previewed and must be explicitly approved before publication. The technical wording must not be attributed to Alexander.

#42039, shipped in 0.153.0, adds a physical-storage mitigation for this issue: when the experimental default-off feature is enabled, eligible cold rollout JSONL files—including shared and forked histories—are zstd-compressed.

That does not prevent the logical write amplification documented here. Repeated replacement_history snapshots and raw tool output may still be written before the cold representation is compressed; #42039 changes how eligible old rollout files are represented, not which payloads are produced or repeated.

Navigation Menu

Repository navigation

Codex session logs grow to 700MB-2GB from repeated compaction history and raw tool output #24948

What version of Codex CLI is running?

What subscription do you have?

Which model were you using?

What platform is your computer?

What terminal emulator and version are you using (if applicable)?

Codex doctor report

What issue are you seeing?

What steps can reproduce the bug?

What is the expected behavior?

Additional information

Impact

What is being duplicated

Source trace

Real-file compression benchmark

Focused patch ready

Release compatibility follow-up

Proposed sequencing

5 remaining items

Failure pattern

Impact

Expected safeguards

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Codex session logs grow to 700MB-2GB from repeated compaction history and raw tool output #24948

Description

What version of Codex CLI is running?

What subscription do you have?

Which model were you using?

What platform is your computer?

What terminal emulator and version are you using (if applicable)?

Codex doctor report

What issue are you seeing?

What steps can reproduce the bug?

What is the expected behavior?

Additional information

Activity

etraut-openai commented on Jun 1, 2026

sriinnu commented on Jun 9, 2026

etraut-openai commented on Jun 9, 2026

banteg commented on Jun 22, 2026

etraut-openai commented on Jun 22, 2026

banteg commented on Jun 22, 2026

sriinnu commented on Jun 23, 2026

sriinnu commented on Jul 8, 2026

atlaslabsworld commented on Jul 13, 2026

runminglu commented on Jul 16, 2026

fengjikui commented on Jul 18, 2026

Impact

What is being duplicated

Source trace

Real-file compression benchmark

Focused patch ready

Release compatibility follow-up

Proposed sequencing

5 remaining items

Akshmit commented on Jul 21, 2026

DAD-Studios commented on Jul 22, 2026

wasulajr commented on Jul 23, 2026

grtninja commented on Jul 23, 2026

Coldain commented on Jul 25, 2026

webhype commented on Jul 26, 2026

yourhousemktg-ctrl commented on Jul 27, 2026

webhype commented on Jul 27, 2026

ducksee commented on Aug 3, 2026

neuralsyndicate commented on Aug 4, 2026

Failure pattern

Impact

Expected safeguards

ewzxyh commented on Aug 8, 2026

georgfranz commented on Aug 10, 2026

dastan-maker commented on Aug 17, 2026

ariccio commented on Sep 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions