Repository navigation
[0.153.4] Rollout migration drops compacted records over 16 MiB; suspected cause of context overflow after resume #44141
Description
What version of the Codex App are you using (From “About Codex” dialog)?
26.903.61454 (bundled CLI/app-server 0.153.4)
What subscription do you have?
Not disclosed (privacy).
What platform is your computer?
macOS
What issue are you seeing?
Summary
The legacy-to-paginated rollout reader discards any individual JSONL record larger than 16 MiB before parsing its type. This also affects compacted records containing replacement_history and guardian_history. Ordinary compaction persistence can write such records without that limit.
Requested action: preserve these replay-critical checkpoints during migration, or fail without publishing the replacement rollout. Add regression coverage for oversized compacted records and resume equivalence. This request follows from the reader/writer mismatch even if the incident's complete causal chain cannot yet be reproduced.
A long-running, image-containing task successfully compacted and continued working, but later failed with context-window errors after reopening. Local inspection found multiple compaction completion markers without corresponding full checkpoint records. The last surviving checkpoint was just below the migration limit.
Evidence boundary: the size filter is confirmed in source. Its responsibility for the missing checkpoints and the subsequent overflow is a strong hypothesis, not a captured before/after migration reproduction. No per-record deletion log or pre-migration snapshot was available.
Environment
Background paginated-rollout migration was enabled; the inspected history was already paginated. The first failure happened before custom context-window overrides were added. Increasing the configured window afterward did not recover the task.
Confirmed source behavior
References are pinned to rust-v0.153.4:
read_rollout_recorduses a 16 MiB per-line limit. An oversized line is drained and returned asline: None, regardless of record type.- The canonicalization loop skips those records. Successful migration then publishes the replacement file.
replace_compacted_historystores the replacement history and auxiliary guardian history in oneCompactedrecord. The ordinary rollout writer serializes and writes the whole record without this size filter.
The reader's size check and discard branch are also present in the main-branch source inspected while preparing this report. This is source inspection, not a runtime test of a newer release.
Sanitized observations from the affected history
The local rollout contained several hundred successful-compaction markers, but the latest group of markers had no corresponding full top-level compacted records. Matching the event sequence, not just comparing aggregate counts, identified this gap.
The last surviving full checkpoint was approximately 15.5 MiB, just below the 16 MiB reader limit. Its replacement history retained many user images, and its auxiliary guardian history also contained image data. Image-URL fields dominated the physical record. Auxiliary images may duplicate main-history images and must not all be counted as main-model input.
After an additional image-containing user record, the next successful-compaction marker lacked its full checkpoint body in the inspected canonical history, as did later completions. The original sizes of the missing checkpoint records cannot be measured from the remaining file.
Compaction was demonstrably effective while the task was running: successful model requests immediately afterward had substantially fewer input tokens than the pre-compaction context. After reopening, a short user message failed with:
Codex ran out of room in the model's context window.
Subsequent remote-compaction attempts also failed. A fork inheriting the affected history failed too. The user-visible error recommended starting a new thread or clearing earlier history.
The UI later displayed a completely full enlarged window, but the associated error record had zero input/output tokens and a total set to the window maximum. That display is not proof of a measured prompt of that exact token count. Disk bytes, image-URL lengths, and model tokens are separate measurements.
Suspected failure sequence
- Compaction succeeds in memory and writes an image-containing replacement-history snapshot.
- A later migration encounters an oversized checkpoint record and omits it from the replacement rollout.
- The live task can continue using its in-memory state, so loss is not immediately visible.
- Resume/fork reconstructs history using an older surviving checkpoint plus subsequent raw records.
- The reconstructed input exceeds the model window, and remote compaction cannot recover it.
The surviving older checkpoint was followed by millions of characters of tool-heavy history before the affected fork's inherited boundary. This supports the replay-inflation hypothesis, but the complete failed server-side prompt and its actual token count were not captured.
What steps can reproduce the bug?
These are proposed tests for the production migration path; they have not been executed as an isolated end-to-end reproduction in this report.
- Build a synthetic legacy rollout with valid session metadata, an earlier checkpoint, intervening tool output, a later valid
compactedrecord larger than 16 MiB, and a trailing small user message. Use placeholder image data, not private screenshots. - Run the real legacy-to-paginated migration against that isolated fixture.
- Assert that migration either preserves the latest checkpoint's effective replacement history or fails without replacing the original file. Continuing past the oversized record while losing it must fail the test.
- Compare resume history before and after migration; superseded tool output must not re-enter effective context because a checkpoint was lost.
- Cover records just below, at, and above the limit, including a record exceeding the limit primarily because of
guardian_history.
The immediate safeguard could be to fail migration without publishing on oversized state-bearing records. Longer-term options include streaming preservation or externalizing large media. Simply increasing the limit would postpone the same class of failure.
What is the expected behavior?
Migration should preserve the effective compacted context needed for resume. If a checkpoint cannot be migrated safely, it should leave the original history intact and report an actionable failure, rather than publish history with the checkpoint omitted.
Compaction completion markers alone should not be treated as proof that the corresponding resumable context is present.
Additional information
Related reports
- Codex session logs grow to 700MB-2GB from repeated compaction history and raw tool output #24948 documents rollout growth and repeated image-containing compaction snapshots.
- Remote compaction v2 retains raw media and can trigger repeated auto-compaction #38835 documents raw-media retention across remote compaction.
This report focuses on the additional persistence failure: a generic per-record migration limit can remove a successfully generated compaction checkpoint and change the history used on resume.
This report was prepared with Codex assistance from local, read-only inspection. Private conversation content, screenshots, credentials, project names, task identifiers, and raw diagnostic files are omitted.
Activity
MonkeyShawn commentedon Sep 10, 2026
Temporary workaround only — recovery verified with CLI/app-server 0.153.4; the underlying bug is not fixed
For others whose task cannot reply or run remote compaction, I recovered an affected task using incremental conversation rollback + a temporary text-only compaction input, followed by a normal-client reload test. This is a temporary way to regain access to a task, not a permanent solution or a fix for the migration behavior reported above. The problem may recur.
The tested boundaries were 5, 10, and 15 recent user turns removed in total. The 5- and 10-turn attempts, including text-only variants, still failed. At 15 turns, native compaction succeeded with the temporary text-only input. 15 is the first successful boundary tested, not a universal number or a proven minimum; 11–14 were not tested. Rollback alone was not established as sufficient.
Recovery procedure
thread/reverttakesthreadIdandbeforeTurnId; the selected turn and everything after it are removed from active history. Count actual user turns, not tool calls or repeated compaction failures. Choose boundaries 5, then 10, then 15 turns back from the original endpoint. This operation changes conversation history; it does not undo workspace file changes. Check the task's model/reasoning settings afterward because revert can restore older settings.input_modalitieschanged to["text"], supplied through the process-localmodel_catalog_jsonoverride to a temporary app-server. Codex's own input normalization replaced model-input images with text markers. Global configuration was unchanged; auxiliary Guardian history was left to the native implementation. This is an advanced, version-specific workaround, not a normal desktop setting. Image details will not survive in active context merely because a marker remains; preserve the originals and selectively reintroduce necessary images later. Advance another five turns only after an explicitcontextWindowExceededfailure; a timeout or network error is not a reason to discard more history. Stop on success or at your chosen rollback limit.thread/compact/startis asynchronous. Wait for the compaction item and turn to complete without error, then confirm a full newcompactedrecord was actually saved. Close the temporary recovery process, reload the task in the normal client with normal settings, and send a small request. This checks that the saved state works after reload, rather than only in the compactor's memory. Avoid concurrent writers while switching clients.For developers implementing step 3–4, the relevant RPCs in the tested version were the following, after normal app-server initialization and thread resume. These are protocol examples, not shell commands; use the schema matching your installed version and a verified backup/boundary:
{"id":10,"method":"thread/revert","params":{"threadId":"<affected-thread-id>","beforeTurnId":"<first-turn-to-remove>"}} {"id":11,"method":"thread/compact/start","params":{"threadId":"<affected-thread-id>"}}The temporary process used the standard
-c model_catalog_json=...override. Keep the rest of the matching catalog intact and end that process after compaction; do not permanently disable image input or replace the catalog with one from a different release.Observed result: the full new checkpoint was 3,774,432 bytes (~3.60 MiB), below the 16 MiB migration-reader threshold. A normal request after reopening succeeded with 69,291 input tokens out of a 258,400-token window. Recent work and pending requests were restored through a short handoff; the original removed text and images remained in local backups. No application source, global configuration, or project code was changed by this recovery.
Alternative if you want to avoid rolling back the task: leave it and its backup intact, start a genuinely new task, and supply only the short handoff plus references to the current workspace and local archive. This preserves access to the original text without requiring the new model context to inherit all of it. Increasing the configured context window did not recover the affected task in this incident.
This result does not prove the full migration-to-overflow causal chain or guarantee that another task will recover at the same boundary. Please treat this only as a temporary recovery option while awaiting an upstream fix; successful recovery is not evidence that the bug is resolved or that recurrence is prevented.