Repository navigation
Long threads are unrecoverable when rollout JSONL grow beyond resume/list limits #25215
Description
Summary
A long-running /goal workflow can create a rollout JSONL that Codex Desktop can no longer safely resume or list, even though the workflow is behaving as designed.
In this case, the thread was running normally under Codex Desktop with WSL enabled. The user paused the active goal to install a new Desktop update, then the updated app could not recover/resume the same thread because the local rollout had grown to about 3.06 GB. Small threads still loaded. The large goal thread repeatedly caused app-server failures and the UI fell back to Codex app-server is not available errors.
This is not just a generic "large file" problem. /goal encourages exactly this usage pattern: an agent runs for a long time, loops through work, compacts context, spawns/reviews sub-work, and keeps appending local history. The model-visible context is bounded by compaction, but the persisted rollout file is not bounded or rotated.
Environment
- Codex Desktop: Windows Store app, observed after update to package family
OpenAI.Codex_26.527.x_x64__2p2nqsd0c76g0 - App-server: WSL/Linux Codex runtime launched by Desktop
- Platform: Windows 11 x64 with Codex Desktop configured to run the agent in WSL
- Subscription/model: ChatGPT-authenticated Codex Desktop, high-reasoning model
- Local state paths, user names, and project names are intentionally redacted
Reproduction shape
- Use Codex Desktop on Windows with WSL-backed execution.
- Start a long-running
/goalworkflow that performs repeated local work, review loops, sub-agent/tool loops, and iterative improvement passes. - Let the goal run for days, including many turns and multiple context compactions.
- Pause the thread to install a Desktop update.
- Reopen Codex Desktop and click/resume the same large thread.
- The app-server crashes or becomes unavailable while hydrating/resuming the thread; a smaller thread still loads successfully.
Local evidence from the affected profile
The affected rollout file:
~/.codex/sessions/2026/05/19/rollout-<timestamp>-<thread-id>.jsonl
size: 3,063,881,398 bytes
A rescue rollout built from the latest compacted checkpoint plus subsequent records was much smaller:
~/.codex/rescue_rollouts/rollout-rescue-<timestamp>-<thread-id>.jsonl
size: 20,737,905 bytes
lines: 382
records: session_meta=1, compacted=1, turn_context=1, event_msg=140, response_item=239
payloads included: thread_goal_updated=61, function_call=82, function_call_output=82, token_count=53, context_compacted=1
The rescue was generated only after a full backup, and only by extracting the latest compaction boundary and following records. That is not a reasonable normal recovery path for a product feature.
Desktop logs around the failing resume showed the large thread pending thread/resume and thread/goal/get, then the app-server becoming unavailable:
app_server_connection.closed code=9 ... transport=stdio
fatal_error_broadcasted ... (code=9, signal=null)
Request failed ... method=thread/resume ... error={"code":-32000,"message":"Codex app-server is not available"}
Request failed ... method=thread/goal/get ... error={"code":-32000,"message":"Codex app-server is not available"}
app_server_restart_recovery_failed ... errorMessage="Codex app-server is not available"
Small-thread control test in the same app/profile loaded successfully, so the failure correlated with the large rollout/resume path rather than all Desktop startup.
Source-level RCA
Current origin/main at 00ca857d3ff6883b7334292d887601344e1bd029 still has full-file rollout hydration in key resume/list paths.
codex-rs/rollout/src/recorder.rs reads the whole rollout into one string, then parses all lines into a Vec:
pub async fn load_rollout_items(
path: &Path,
) -> std::io::Result<(Vec<RolloutItem>, Option<ThreadId>, usize)> {
trace!("Resuming rollout from {path:?}");
let text = tokio::fs::read_to_string(path).await?;
...
let mut items: Vec<RolloutItem> = Vec::new();
for line in text.lines() {
... serde_json::from_str(line) ...codex-rs/thread-store/src/local/read_thread.rs attaches history by loading the full rollout when include_history=true:
let items = load_history_items(&path).await?;
thread.history = Some(StoredThreadHistory { thread_id, items });load_history_items() delegates back to the full-file rollout loader:
let (items, _, _) = RolloutRecorder::load_rollout_items(path).await?;codex-rs/app-server/src/request_processors/thread_processor.rs resumes stored threads with history included:
.read_stored_thread_for_resume(thread_id, path, /*include_history*/ true)The thread/turns/list path also has a source comment acknowledging the scalability problem:
// This API optimizes network transfer by letting clients page through a
// thread's turns incrementally, but it still replays the entire rollout on
// every request. Rollback and compaction events can change earlier turns, so
// the server has to rebuild the full turn list until turn metadata is indexed
// separately.That design makes a 3 GB goal-created rollout inherently unsafe to hydrate. Even if context compaction keeps model context manageable, the local JSONL keeps growing and later resume/list paths still eagerly read and replay it.
Expected behavior
A /goal thread should remain recoverable after an app update if it was running normally before the update.
At minimum:
- Long-running
/goalworkflows should not create local session files that the product cannot reopen. - The app should not need to read a multi-GB rollout into one string to resume a thread or list turns.
- If a thread is too large, Codex should show a recoverable per-thread error and offer a built-in compact/export/rescue path.
- Goal state should be exportable/restorable from the latest compacted checkpoint without manual SQLite edits or JSONL surgery.
- Auto-update should preflight active goal/session size and warn or create a safe checkpoint before replacing the running app.
Actual behavior
- The active goal thread became unrecoverable from Desktop after updating.
- Resuming the large thread made the app-server unavailable while smaller threads loaded.
- Recovering the work required unsupported manual steps: backing up the profile, finding the latest compaction boundary in a 3 GB JSONL, creating a reduced rescue rollout, and redirecting local thread metadata.
- The recovered handoff is necessarily lower fidelity than a normal
/goalresume because old pre-compaction records had to be dropped to keep the app usable.
Suggested fixes
- Stop using
read_to_stringfor rollout hydration on resume/list paths. Stream parse JSONL, enforce record/byte caps, and avoid building a fullVec<RolloutItem>unless explicitly exporting. - Index turn metadata separately so
thread/turns/listcan page without replaying the full rollout on every request. - Make context compaction also produce a storage checkpoint. After a successful compaction, Codex should be able to prune or archive pre-compaction records while preserving an exportable full archive.
- Add rollout rotation or hard warnings for
/goalsessions. Example thresholds: 250 MB warning, 500 MB danger, 1 GB force checkpoint/export path. - Add a built-in "recover from latest compaction" command that creates a continuation thread or handoff without requiring direct DB edits.
- During Desktop update, detect active goal threads and write a verified resume checkpoint before replacing the app/runtime.
- Isolate per-thread failures. One oversized thread should not make the whole app-server or WSL-backed Desktop unusable.
Related issues
- # Codex desktop main-process crash:
RangeError: Invalid string lengthwhen loading sessions whose rollout JSONL exceeds V8's max string length #22004: Desktop main-process crash when rollout JSONL exceeds V8 max string length - Codex app can freeze with very large rollout/history JSONL files #22991: app freezes with very large rollout/history JSONL
- Codex Desktop becomes unusable on long active threads due to app-server/renderer memory and TRACE log churn #21134: long active thread caused multi-GB app-server memory footprint
- app-server loads ALL session files on every thread/list call — causes high CPU, slow startup, and background token waste #22411: app-server loads/deserializes all session files for thread/list
- Codex Desktop high CPU from unbounded active thread metadata and local history/list processing #24510: long-running goal sessions emit many goal/progress events and stress local history/list paths
- Codex Desktop remote compaction repeatedly fails on long sessions #24544: long sessions break
/goalworkflows via compaction failures - in goal last steer is re-included after each compaction #21291:
/goaland compaction behavior issue - /goal long-running loop produces 480 KB single log lines via chain-nested turn{} tracing spans (34 GB log in one day) #23340:
/goallong-running loop caused runaway log growth - Windows Desktop WSL app-server fails to launch due to CRLF/LF SQLx migration checksum mismatch #23777: separate Windows Desktop WSL update failure that triggered this recovery attempt
- Desktop update prompt should show target versions, component diff, and environment impact #23053: update prompt should surface target versions/environment impact before users accept risky updates
Privacy note
I cannot attach the raw 3 GB rollout because it contains private local conversation history and project content. The sizes, record counts, method names, source paths, and stack-level RCA above are sanitized.
Activity
github-actions commentedon May 30, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
92645417d9e5c763259dbebc306e3e commentedon Jun 3, 2026
Adding one narrower observation from the CLI/TUI and remote-control side.
This seems to be the same root cause as the issue describes: app-server still lacks a bounded rollout resume/read strategy for very long JSONL histories. Even when model-visible context is bounded by compaction, local resume/read paths can still effectively behave as if the whole durable transcript must be materialized before the client becomes usable.
This is especially painful on display/performance-constrained clients, such as Codex remote control from the ChatGPT iPhone app. Those clients are much less tolerant of large app-server payloads, long transcript hydration, and repeated replay work than a desktop machine. A thread that is technically recoverable on desktop can still become practically unusable on mobile/remote-control surfaces.
In my testing/design review, the minimal safe direction seems to be:
SessionMeta;Compacted(replacement_history=Some(...))checkpoint;The user-visible impact is severe in long sessions. In particular, message withdraw/edit flows become almost unusable: after cancelling or withdrawing a pending message, the client appears to reload/replay a very long transcript again, causing a long stall every time. This is a frequent operation in long-running sessions, so the cost is not just initial resume latency; it affects normal interactive editing. On iPhone remote control, this kind of repeated hydration/replay cost is even more noticeable.
So I think the core missing primitive is not only paged UI rendering, but a bounded app-server/thread-store contract for “resume from latest valid compaction checkpoint + suffix” while preserving the same model-visible context as the full rollout replay.
Generated by Codex CLI.
[-]/goal can make Desktop threads unrecoverable by growing rollout JSONL beyond resume/list limits[/-][+]Long threads are unrecoverable when rollout JSONL grow beyond resume/list limits[/+]0xdevalias commentedon Jun 17, 2026
Partially impacted by
rust-v0.140.0: #27031 avoids rereading and reparsing the same rollout history during coldthread/resume, with the reported savings growing with rollout size.That should reduce one resume hot-path cost for long rollouts. It does not implement bounded/streaming resume, rollout rotation, pruning, or recovery from multi-GB JSONL files, so this issue is probably not resolved. Current builds may just be less expensive in the cold-resume case.
linear commentedon Jul 8, 2026
from fc:
truongvknnlthao-gif commentedon Jul 21, 2026
I reproduced this on macOS and confirmed that the legacy-thread resume cost remains on current
main.Environment
0.145.0-alpha.18fd3c1dc13d0a0941af406e1bc1f697c9d14110eahistory_mode=legacyI exercised the same request shape used by the desktop tail-loading path:
All probes used isolated
CODEX_HOMEdirectories and read the original rollouts without modifying the real app database or session files.fd3c1dcfd3c1dcSo pagination bounds the response sent to Desktop, but does not bound legacy rollout replay/model-history reconstruction. In the 1.016 GB case, returning only a 710 KB initial page still required about 1.36 GiB peak server RSS on current main.
I also ran a controlled causality experiment on copies only:
session_metaplus the latest compaction checkpoint and its suffix produced a 50.2 MB recovery copy.data:imageURLs totaling 49,874,404 bytes.A second 217 MB rollout showed the same image effect: its latest-checkpoint copy was 564 KB; replacing one 361,102-byte inline image reduced it to 203 KB, and the bundled core resumed it at 57 MiB peak RSS.
This supports two independent costs:
The recovery copies are intentionally lossy and are not a proposed upstream fix: they omit old visible history and replace image data. A production fix likely needs a data-preserving streaming/indexed projection or migration for legacy rollouts, correct rollback/compaction semantics, bounded or externalized inline binary payloads, and regression tests asserting that peak RSS does not scale with the full legacy JSONL for a bounded initial page.
I cannot share the raw rollouts because they contain private conversation and project content, but I can provide the probe method and aggregated metrics if useful.
DataDrivenDNA commentedon Aug 13, 2026
I can reproduce the legacy
thread/turns/listcost and found a blocker in the new legacy-to-paginated migration that affects exactly the oversized sessions this issue tracks.Sanitized field evidence
thread/turns/listrequests taking 5.4-7.1 seconds each, about 65.6 seconds in aggregate.Current
mainata7b8c074b577f897111c14de3a5e127b91e2a479confirms the legacy behavior:thread_turns_list_response_innerpages the response but replays the complete rollout on every request. The indexed/paginated path added in #37175 and #37348 is the right architectural direction.Migration causality test
I copied only this rollout into an isolated
CODEX_HOMEand ran the published@openai/codex@0.148.0-alpha.12command against the copy:The command reported
status: migrated, but the result changed as follows:So the migration silently discarded every oversized compaction and rewound the latest retained checkpoint by roughly 14.5 hours. The real rollout was not modified.
This matches the source.
read_rollout_recorddrops any record overMAX_ROLLOUT_LINE_BYTESby returningline: None. The existing regression test covers only an oversizedfunction_call_outputand expects it to be dropped. However,LegacyRolloutCanonicalizerotherwise preservesRolloutItem::Compacted, because it is model-context state rather than disposable UI history.Proposed correctness contract
Before automatic migration is enabled for large legacy sessions:
migratedafter silently dropping an oversized compaction or other model-context-bearing record.list_turnsand do not scale with the full legacy JSONL size.The repository contribution policy says external PRs require an explicit maintainer invitation. If this contract matches the intended migration behavior, I am happy to prepare the focused failing test and implementation in an invited PR.
92645417d9e5c763259dbebc306e3e commentedon Aug 13, 2026
I checked my own pre-migration backup to determine whether the 16 MiB record limit had caused silent data loss in my sessions.
The backup was decrypted and extracted into an isolated temporary directory. It was not imported into the active Codex home. I then scanned every rollout JSONL as raw bytes, measuring complete newline-delimited records without retaining or reporting transcript contents.
Results
compactedrecordcompactedrecords scannedThe largest record was a
response_item/function_call_output, not a compaction record. The largest compaction was approximately 6.25 MiB, well below the 16 MiB migration limit.I also checked the limited
/goalactivity in this backup:thread_goal_updatedrecordsOnly two rollout files contained
thread_goal_updatedevents, and neither came close to the limit.Interpretation
This particular backup does not appear to have been affected by the oversized-record loss described above. However, the result also confirms that checking only the migrated paginated files would not be sufficient: a successful lossy migration would already have removed every oversized source record. The meaningful check must use a pre-migration backup or snapshot.
The current behavior still appears unsafe in the general case because the 16 MiB check is performed before the record is classified. An oversized semantic record such as
compacted,turn_context,world_state, or a model-visible message can therefore be skipped through the same path as an oversized tool output.I agree with the proposed fail-closed contract: if an oversized record cannot be preserved, migration should report failure and leave the legacy rollout unchanged rather than publish a paginated replacement with missing semantic state.
This report was organized with Codex CLI from locally computed size and type metadata. Local paths, storage device names, session contents, prompts, tool outputs, credentials, and repository details have been omitted.