Paginated history drops valid flattened rollout records and reuses ordinals

What version of Codex CLI is running?

Observed on 0.146.0-alpha.10.1. The affected source paths remain unchanged in rust-v0.146.0-alpha.14.

What platform is your computer?

Linux x86_64.

What issue are you seeing?

Paginated rollout history has inconsistent RolloutLine decoding. Some readers deserialize JSON directly into RolloutLine, while the canonical loader first parses into serde_json::Value and then calls serde_json::from_value::<RolloutLine>.

A serialized RolloutLine::EventMsg(EventMsg::TokenCount(...)) containing populated rate-limit and credit data is valid JSON and succeeds through the value-first path, but direct serde_json::from_str::<RolloutLine> / serde_json::from_slice::<RolloutLine> rejects it.

Two paginated-history readers currently use the direct path:

codex-rs/rollout/src/ordinal.rs: scan_next::<RolloutLine>()
codex-rs/thread-store/src/local/thread_history_materialization.rs: serde_json::from_slice(line_bytes)

This produces two related failures:

Resume ordinal discovery silently skips the valid final record, steps back to an earlier ordinal, and can append a new record with an already-used ordinal.
SQLite history projection silently skips the valid record while advancing its byte offset beyond it, leaving the materialized projection behind the canonical JSONL permanently.

Once affected, catch-up can begin with one replayed boundary ordinal equal to projection_state.next_ordinal - 1; rejecting that boundary prevents the remaining valid suffix from materializing.

What steps can reproduce the bug?

Add a focused test that serializes a paginated RolloutLine containing a token-count event with rate-limit and credit fields:

let encoded = serde_json::to_string(&RolloutLine {
    timestamp: "2026-07-09T00:00:05Z".to_string(),
    ordinal: Some(5),
    item: RolloutItem::EventMsg(EventMsg::TokenCount(TokenCountEvent {
        info: Some(TokenUsageInfo::full_context_window(1_000)),
        rate_limits: Some(RateLimitSnapshot {
            limit_id: Some("limit-1".to_string()),
            limit_name: None,
            primary: Some(RateLimitWindow {
                used_percent: 1.0,
                window_minutes: Some(60),
                resets_at: Some(1),
            }),
            secondary: None,
            credits: Some(CreditsSnapshot {
                has_credits: true,
                unlimited: false,
                balance: Some("1".to_string()),
            }),
            individual_limit: None,
            spend_control_reached: None,
            plan_type: None,
            rate_limit_reached_type: None,
        }),
    })),
})?;

assert!(serde_json::from_str::<RolloutLine>(&encoded).is_err());
assert!(
    serde_json::from_str(&encoded)
        .and_then(serde_json::from_value::<RolloutLine>)
        .is_ok()
);

Then:

Append that record as the final line of a paginated rollout and resume it. Ordinal discovery ignores ordinal 5.
Materialize the same rollout into the thread-history SQLite projection. The line is rejected, but the stored byte offset advances past it.

What is the expected behavior?

Every rollout reader should use the same value-first decoding semantics as the canonical loader. Resume should continue after the real final ordinal, and SQLite materialization should project every valid canonical record.

For projections already affected by the old decoder, narrowly accepting and skipping one replayed first boundary ordinal (next_ordinal - 1) allows the remaining suffix to catch up without rewriting canonical JSONL.

Additional information

rust-v0.146.0-alpha.14 changes nearby thread-list behavior for missing rollout paths, but ordinal.rs and thread_history_materialization.rs still use direct RolloutLine deserialization.

i traced this to the two direct RolloutLine decode sites called out here: reverse ordinal discovery and thread-history materialization both bypass the value-first decode path used by canonical rollout loading.

local tested fix: make those readers parse the physical JSON line as serde_json::Value first, then decode the RolloutLine, preserving the existing behavior of skipping malformed physical lines.

validation on my branch:

just fix -p codex-rollout
just fix -p codex-thread-store
just test -p codex-rollout (113 passed)
just test -p codex-thread-store (158 passed)

because docs/contributing.md says external PRs are invitation-only, i did not open a PR. branch is ready on my fork if maintainers want one: https://github.com/erichanwang/codex/tree/fix/paginated-rollout-value-decode

Confirmed on ChatGPT/Codex for Windows.

Feedback ID: 019fb9de-3796-71e2-b216-b709023f8dbe
Codex app: 26.727.6591.0
CLI/app-server: 0.146.0-alpha.9.2

The raw rollout remains valid with 2,200+ continuous ordinals, but the UI history projection is frozen immediately before ordinal 20. Logs repeatedly report:

invalid type: map, expected f64
thread history projection ... expected ordinal 20, got 21

Impact:

Reopening the task lost the visible history twice.
The UI showed only one user and one assistant message.
Two attempts to branch from a later turn failed.
Restarting Codex did not repair the projection.

Current-session diagnostics were included with the feedback submission; browser-tab logs were excluded.

Two additional Codex for Windows tasks now reproduce the same projection failure on the same installed build:

App: 26.727.6591.0
CLI/app-server: 0.146.0-alpha.9.2

Additional affected task 1

Thread/Feedback ID: 019fc3d5-e9dd-7f51-be4d-8557ffb5a2bd
At inspection time, the canonical rollout contained 155 valid JSONL records with continuous ordinals 0-154, no malformed JSON, and no gaps or duplicate ordinals.
SQLite thread_history_projection_state remained frozen at byte offset 235257, next ordinal 17, with only four projected items.
Logs repeatedly report:
- invalid type: map, expected f64
- thread history projection ... expected ordinal 17, got 18

Additional affected task 2

Thread/Feedback ID: 019fbf98-130c-7e62-b16e-038a76c60401
The canonical rollout contains 432 valid JSONL records with continuous ordinals 0-431, no malformed JSON, and no gaps or duplicate ordinals.
SQLite projection is frozen at byte offset 739950, next ordinal 17, again with only four projected items.
Logs repeatedly report the same map/f64 decoding error followed by expected ordinal 17, got 18.

For both tasks, the app-level read-only history call also fails with:

paginated threads do not support thread/read(includeTurns=true)

User-visible result: reopening either task displays only the initial user input and one assistant response despite the intact canonical rollout.

Negative control: in the same app session, the user reopened two other threads under another project and both appeared to load normally. This suggests the defect is not triggered visibly for every task, even on the same client installation.

Impact now extends beyond missing UI history: substantial weekly model usage is being consumed diagnosing the client and reconstructing work that should remain visible.

Please ensure the eventual repair handles already-stalled projections and the replayed boundary ordinal, rather than only correcting decoding for newly created projections.

A fourth affected task reproduces the same decoder/projection failure and adds a potentially important cross-device trigger.

Thread/Feedback ID: 019fc4f1-b75b-70c0-a8d5-c02fbc9c0003
Feedback has been submitted with current session logs.
Windows host app: 26.727.6591.0
CLI/app-server: 0.146.0-alpha.9.2

Cross-device sequence

The task was actively running on the Windows host, where the complete conversation remained visible in the originating client.
At approximately 8:20 PM ET, the same active task was opened remotely in Codex on a MacBook.
The MacBook displayed the original input and the currently streaming reasoning/thoughts, but none of the completed intervening turns.
The Windows client continued showing the complete in-memory conversation until the user navigated away from that task.

Read-only diagnostics on the Windows host

Canonical rollout: 165 valid JSONL records
Malformed JSON: 0
SQLite thread_history_projection_state: byte offset 439027, next ordinal 17
Only four projected items, at ordinals 9, 10, 12, and 15
Ordinal 17 is a token_count event whose rate_limits value is the structured map already implicated in this issue
Logs report:
- invalid type: map, expected f64
- thread history projection ... expected ordinal 17, got 18
The app-level history call fails with:
- paginated threads do not support thread/read(includeTurns=true)

The first matching projection errors were logged at approximately 8:20:16 PM ET, closely matching the remote-open event. This suggests that cross-device viewing of an active task may initiate the paginated history materialization path and expose the decoder defect, while the originating client temporarily retains a complete live/in-memory view.

The remote view did not corrupt the canonical rollout, but once the originating client navigates away, its complete in-memory representation is expected to be replaced by the same stalled projection. In practical terms, the raw JSONL being recoverable does not eliminate the user impact: reconstructing the missing turns, ordering, tool outputs, and conclusions still consumes substantial time and model usage.

I can confirm this on Linux x86_64 with Codex CLI/app-server 0.147.0. It happened in one of my long paginated threads that I use through both Remote Control and CLI resume.

I inspected my original rollout JSONL and found three fully preserved occurrences of the same sequence:

Reused ordinal	Final record before resume	Newly appended record
`14368`	`event_msg/token_count`	`event_msg/thread_settings_applied`
`17761`	`event_msg/token_count`	`event_msg/thread_settings_applied`
`17809`	`event_msg/token_count`	`event_msg/thread_settings_applied`

In all three cases, the token_count record contains populated structured rate_limits and credits fields. The following thread_settings_applied record then reuses exactly the same ordinal.

After I repaired those three duplicates and rebuilt the projection, the same problem occurred again in the same thread under 0.147.0:

token_count(43706)
thread_settings_applied(43706)

For this recurrence, the paginated projection stopped at:

next_rollout_ordinal = 43707
next_rollout_byte_offset = 141901165

The next physical record at that offset still had ordinal 43706, so the app-server repeatedly reported:

expected ordinal 43707, got 43706

What I observed as a user:

Remote Control could still connect to my host and list the affected thread.
Opening or resuming this particular thread failed or appeared as a generic connection/loading error.
Other threads remained accessible.
The canonical JSONL continued growing, but the SQLite projection stayed frozen.
codex doctor --summary reported healthy authentication, WebSocket connectivity, app-server state, and SQLite integrity.
Restarting alone did not repair the affected projection.

Based on my evidence, this matches the resume ordinal-discovery failure described in this issue more closely than a simple concurrent-writer explanation. A sequential resume appears sufficient: when the final structured token_count is rejected by direct RolloutLine deserialization, ordinal discovery steps back and the resumed writer reuses the previous ordinal.

The current main implementation also still appears to use:

scanner.scan_next::<RolloutLine>()

while ReverseJsonlScanner::finish_record() directly calls:

serde_json::from_slice::<T>(...)

The projection reader received value-first and malformed-rollout resilience changes through #36188, but resume ordinal discovery still appears vulnerable. Once a duplicate has been written, the projector also still rejects it instead of recovering the remaining valid suffix.

I also encountered the related Remote/writer-handoff behavior discussed in #37403, #37450, and #37552. Those issues seem relevant to how the problem is triggered and presented through Remote Control, but the persisted duplicate ordinals in my rollout match #35746 specifically.

I have a sanitized audit, Doctor summary, timestamps, hashes, and a read-only reproduction script available. I am not attaching my complete rollout publicly because it contains private conversation and tool history.

Independent current reproduction on Linux x86_64 / VS Code Web with Codex CLI/app-server 0.147.0.

Thread/Feedback ID for internal trace lookup: 019fe127-1fbf-7423-92d4-98d1f9405449.

The canonical paginated rollout is valid JSONL, but it contains two preserved resume-boundary ordinal reuse sequences:

event_msg/token_count(30091)
event_msg/thread_settings_applied(30091)
event_msg/task_started(30092)

and later:

event_msg/token_count(41264)
event_msg/thread_settings_applied(41264)
event_msg/task_started(41265)

The first duplicate permanently stalled the SQLite history projection at:

next_rollout_ordinal = 30092
next_rollout_byte_offset = 169588767

Every later live projection attempt reports:

thread history projection ... expected ordinal 30092, got 30091

This repeated under two successive app-server processes, so restarting the affected task/app-server did not repair it.

At inspection time, the raw rollout still contained 41,517 records (about 307 MB) and continued through ordinal 41,514. Thus more than 11,000 later physical records remained preserved in canonical storage but were absent from the paginated projection and effectively disappeared from the reopened UI history.

This confirms that stable 0.147.0 remains affected in both parts described here:

Resume ordinal discovery can reuse the final token_count ordinal.
An already affected projection cannot recover when catch-up starts with next_ordinal - 1.

The user-visible missing-history symptom and side/fork failures are the same failure class as #38248. Fixing only new decoding is insufficient; existing affected projections also need a supported rebuild or the narrow replay-boundary recovery described in this issue.

No project name, repository data, prompts, local paths, host/user identifiers, process identifiers, or message contents are included.

Recovery outcome from the same sanitized reproduction:

Before recovery: the projection was frozen at byte offset 169588767 / next ordinal 30092, and exposed 59 turns.
Canonical storage still contained the later records.
After backing up both stores, removing only the two duplicated resume-boundary metadata records, and rebuilding the derived projection, it advanced to the exact rollout EOF at byte offset 306891319 / next ordinal 41514.
thread/turns/list now returns all 79 turns in one page with no projection error.
SQLite quick_check is ok, and the repaired ordinal stream has no gaps or non-increasing values.

This recovered 20 previously invisible turns without reconstructing user or assistant messages. It confirms the missing history was a stale derived projection, not loss of the underlying conversation.

The manual repair required a full backup and unsupported local storage surgery. The product still needs a built-in rebuild/recovery path for already affected projections.

No paths, project or repository names, prompts, message contents, host/user identifiers, or process identifiers are included.

need fix

Third observed incident in the same sanitized Linux x86_64 / VS Code Web thread with CLI/app-server 0.147.0.

Thread/Feedback ID: 019fe127-1fbf-7423-92d4-98d1f9405449.

The failure returned after each successful manual recovery:

First incident: duplicate resume boundaries at ordinals 30091 and 41264. Recovery advanced the projection to EOF 306891319 and next ordinal 41514.
Second incident: later duplicate boundaries appeared at ordinals 47173 and 49862. The projection stopped at byte 380474094 and next ordinal 47174, while canonical storage grew to 419985119.
Third incident: after the second repair had reached exact EOF, a later resume appended this sequence:
- event_msg/token_count(51563)
- event_msg/thread_settings_applied(51563)
- event_msg/task_started(51564)

The projection remained at the previous repaired EOF 419985119 and next ordinal 51564, while canonical storage grew to 457057512. The app-server repeatedly logged:

expected ordinal 51564, got 51563

For the third recovery, removing only the duplicate metadata record and running the normal projection catch-up restored:

exact EOF: 457057512
next ordinal: 54580
continuous ordinals: 0-54579
thread/turns/list: 93 turns, no next cursor, no projection error
SQLite quick_check: ok

No user or assistant message required reconstruction. A successful repair is therefore not durable: a later sequential resume in the same thread can create another duplicate and freeze the projection again.

The product needs both:

corrected resume ordinal discovery when the previous record is a structured final token_count
an automatic recovery path for an existing next_ordinal - 1 replay boundary

This report contains no private paths, project or repository names, prompts, message contents, host or user identifiers, or process identifiers.

Independent confirmation on stable Codex CLI/app-server 0.147.0, Linux x86_64, from two long paginated threads.

Both canonical rollouts contained the resume-boundary ordinal reuse described in this issue:

event_msg/token_count(N)
event_msg/thread_settings_applied(N)
event_msg/task_started(N + 1)

The duplicated ordinals were 1771 in one rollout and 6329 in the other. Both duplicate boundaries occurred immediately after a structured token_count record.

User-visible behavior:

resuming opened a much older projected prefix instead of the recent conversation;
the later conversation remained present in the canonical JSONL;

forking failed during TUI bootstrap with:

thread/fork failed: failed to prepare paginated fork:
thread-store internal error: thread history projection ...
expected ordinal 1588, got 1611;
1 rejected rollout lines cannot cover that gap

restarting did not repair the projection.

These two incidents also contained NUL-only physical JSONL records at abrupt host-restart boundaries: two in the first rollout and one in the second. The SQLite database itself passed its integrity check, but its projection cursor no longer described a usable continuation of the rollout. This produced a second failure shape beyond the simple next_ordinal - 1 replay: the saved byte offset landed inside a later physical record, so one rejected fragment was followed by an ordinal gap that the projector treated as fatal.

Recovery result after backing up both stores:

removed only the three NUL-only physical records and the earlier record from each duplicated ordinal pair;
rebuilt only the derived thread-history projection;
first rollout projected to exact EOF at byte offset 65,679,285, next ordinal 11,354, exposing 96 turns;
second rollout projected to exact EOF at byte offset 86,424,395, next ordinal 10,152, exposing 130 turns;
SQLite integrity remained ok;
no user or assistant messages were reconstructed.

This confirms that stable 0.147.0 can preserve the recent conversation in canonical storage while resume/fork becomes permanently stuck on the derived projection. Fixing the two inconsistent decode sites prevents new ordinal reuse, but already affected projections also need an automatic rebuild/recovery path when the saved byte boundary or ordinal invariant no longer matches the rollout.

No rollout files, prompts, message contents, private paths, project names, host/user identifiers, or thread IDs are included.

Independent current reproduction on Linux x86_64 with a paginated thread created by stable Codex CLI/app-server 0.147.0 (not migrated from 0.146.x).

The stale resume was visible in both Codex CLI and the ChatGPT desktop Codex surface using the shared local history state. Environment details:

external Codex CLI/app-server: 0.147.0
ChatGPT desktop package: 26.803.81509-1
Codex bundled with the desktop app: 0.147.0-alpha.6.6

Sanitized measurements from the affected thread:

canonical rollout: 121,953,091 bytes, valid JSONL, tail ordinal 10802
projection frozen at byte 81,373,381 with next ordinal 7328
the record at the frozen boundary was thread_settings_applied(7327), following token_count(7327)
a second resume-boundary duplicate existed later: token_count(9090) followed by thread_settings_applied(9090)
24 turns and 32 user messages after the frozen cursor were still present in the canonical rollout

The app-server repeatedly logged:

thread history projection ... expected ordinal 7328, got 7327

This is not only unrepaired state originating in 0.146.x: the session metadata records cli_version: 0.147.0 at creation, and the duplicate boundaries were appended during later resumes under 0.147.0. This is also a concrete counterexample to the "pure 0.147.0 threads were clean" observation in #38792.

Recovery exposed an additional self-healing requirement. After aligning the first replayed boundary, projection advanced to the second duplicate but rolled the transaction back. A one-boundary tolerance is therefore insufficient when one unprojected suffix contains multiple replayed boundary ordinals. I recovered it by committing projection up to the second boundary, aligning that boundary, then projecting the remainder.

Post-recovery verification:

projection cursor: 121,953,091 (exact EOF)
next ordinal: 10803
PRAGMA quick_check: ok for both state and thread-history databases
canonical rollout hash unchanged

Please treat the writer bug and recovery path as shared app-server/history-store behavior rather than CLI-UI-only. Automatic repair needs to handle multiple replayed boundary ordinals in a single suffix.

Second independent recurrence on exact codex-cli 0.147.0, after repairing the projection to an exact JSONL EOF:

Offline repair finished at byte 6,937,927, next ordinal 12302, with PRAGMA quick_check = ok.
The very next resume later wrote this physical boundary:
- ordinal 12408: populated token_count
- ordinal 12408: thread_settings_applied
- ordinal 12409: task_started
The projection then stopped at byte 7,289,327 with next expected ordinal 12409, because the next physical record was the duplicate 12408.
The canonical JSONL remained completely valid and newline-terminated. Its size reached 12,125,753 bytes, so 4,836,426 bytes (including 19 assistant messages) were present on disk but invisible after reopening.
The later abrupt reboot did not create the duplicate: the duplicate was written at resume, well before that reboot. The reboot merely exposed the stale materialized history.

I also reproduced the writer failure directly from the rust-v0.147.0 tag with a focused test whose final durable line is a populated flattened token_count at ordinal 5:

unpatched 0.147.0, two runs: resulting ordinals [0, 5, 1]
with scan_next::<serde_json::Value>() followed by serde_json::from_value::<RolloutLine>(): [0, 5, 6]
full codex-rollout crate: 115/115 tests passed; scoped Clippy passed

This confirms that a clean projection repair is immediately undone by the first affected resume on 0.147.0, and that the value-first ordinal decode fixes the reproducer. Existing affected projections still need the narrow boundary catch-up described in the issue.

New 0.150.1 recurrence: `task_started` itself reuses the terminal `token_count` ordinal

This is another recurrence in a previously recovered long-running paginated task. The current duplicate was created after the earlier recovery. No rollout, SQLite state, or process was modified during this evidence capture.

Environment

Windows 11 ARM64 Desktop client: 26.825.4187.0
Desktop-bundled runtime: 0.150.0-alpha.12.2
Remote Linux x86_64 app-server: 0.150.1
Session originally created by 0.147.0
History mode: paginated

Exact stalled boundary

The canonical rollout currently contains 707,757,918 bytes and 79,078 physical lines. The projection state is frozen at:

next_rollout_byte_offset = 619432687
next_rollout_ordinal     = 70387

The exact valid JSON records at the boundary are:

line 70392  offset 619428701  ordinal 70385  response_item/custom_tool_call_output
line 70393  offset 619431873  ordinal 70386  event_msg/token_count
line 70394  offset 619432687  ordinal 70386  event_msg/task_started   <-- reused
line 70395  offset 619432936  ordinal 70387  turn_context
line 70396  offset 619433818  ordinal 70388  response_item/message

This differs from the previously common signature:

token_count(N)
thread_settings_applied(N)
task_started(N+1)

Here the new task_started record directly reuses the prior token_count ordinal. The token-count record was written at 2026-08-26T23:25:12.228Z; the duplicated task start was written at 2026-08-27T07:09:57.713Z.

A complete read-only ordinal scan found exactly one duplicate and no gaps among parseable records. Six older malformed/non-JSON physical lines exist earlier in the file, but the projector had already advanced past all of them. They do not explain this later stop.

Preserved but invisible answer

The rollout continued normally through ordinal 79070. The newest persisted turn contains:

a user message;
reasoning and tool events;
an assistant message with phase=final_answer;
a final token-count record;
event_msg/task_complete at 2026-08-28T11:05:06.141Z.

The thread reports idle, so that turn is terminal. However, Desktop and the native task-read projection still expose only older turns. The unprojected suffix is 88,325,231 bytes, approximately 84.2 MiB, covering about 8,684 ordinals.

The live writer logs the same failure for every later persist and final flush:

thread history projection expected ordinal 70387, got 70386

Therefore, the missing answer is not model failure or transcript deletion. It is present in the durable rollout and hidden by the stalled derived projection.

Impact

A completed answer is invisible in Desktop.
Native task reads return stale lifecycle/history.
Every later append repeats projection warnings.
Further work can continue in the canonical rollout while UI/API consumers remain frozen on older state.
A user may resend work that already completed.

Fix direction

Ordinal allocation for a new turn must use the last physically valid rollout ordinal even when the terminal record is a structured token_count.
task_started must never reuse an existing ordinal, even if resume-time metadata is not emitted.
The projector should recover or expose a bounded corruption error instead of permanently freezing every later valid record.
Add a regression with terminal structured token_count(N) followed directly by a later task_started; assert that the latter is N+1.
Expose projection lag in Desktop so a preserved final answer is not presented as absent.

This extends the current-main reverse-reader evidence already documented above: the ordinal-reuse bug is still reproducible on stable 0.150.1, and thread_settings_applied is not required for the bad boundary.

Independent macOS Desktop recurrence in two threads at the same recovery boundary

Adding a macOS confirmation from an unmodified Codex Desktop installation. No rollout or SQLite file was modified before evidence capture.

Environment

macOS 26.6.2 (25G83), Apple Silicon (arm64)
ChatGPT/Codex Desktop 26.825.32147 (build 7303)
Desktop-bundled runtime codex-cli 0.150.0-alpha.12.2
Paginated history

Two independent affected threads

Both rollouts were valid JSONL and continued receiving newer records, but their SQLite history projections stopped at the duplicate boundary.

Thread A:

2026-08-28T07:46:45.015Z  9013  event_msg/token_count  primary.used_percent = 3.0
2026-08-28T07:47:36.603Z  9013  event_msg/thread_settings_applied
2026-08-28T07:47:36.609Z  9014  event_msg/task_started

Thread B:

2026-08-28T07:47:14.597Z  8308  event_msg/token_count  primary.used_percent = 3.0
2026-08-28T07:47:36.819Z  8308  event_msg/thread_settings_applied
2026-08-28T07:47:36.823Z  8309  event_msg/task_started

The two reused ordinals were written only 216 ms apart, which is consistent with a shared Desktop/app-server recovery or reload boundary rather than unrelated content failures.

A read-only scan of 616 local rollout JSONL files (about 36.9 GB total) found:

zero JSON parse errors;
exactly three duplicate-ordinal points;
two were the current failures above;
the third was an older instance of the same boundary pattern that had already been recovered.

Projection impact and bounded recovery

After taking an online SQLite backup, I advanced only each affected projection byte cursor past the duplicated no-body thread_settings_applied record. I did not edit either canonical JSONL.

Thread A: 34 -> 46 turns; 2803 -> 4206 projected items
Thread B: 53 -> 64 turns; 2834 -> 3460 projected items

Both projections then caught up exactly to JSONL EOF, PRAGMA quick_check returned ok, and the newer turns became readable again.

This independently matches the reverse-reader diagnosis already documented here: a terminal structured token_count whose used_percent is serialized lexically as 3.0 is skipped during resume ordinal discovery, so the resume event reuses the previous ordinal. The local evidence confirms the failure still affects the current macOS Desktop runtime generation and can hit multiple threads at the same recovery boundary.

I ran into this exact failure on Linux: the rollout JSONL was intact, but codex resume kept loading an old projection after an ordinal was reused at a resume boundary.

I packaged the recovery procedure I used into a small MIT-licensed tool:

https://github.com/mulfyx/codex-session-repair

It starts with read-only diagnosis and only offers a repair for three narrow, tested boundary shapes involving a duplicated thread_settings_applied or task_started after token_count. Before changing anything, it requires the target session to be closed, creates a private backup, verifies SQLite integrity, and checks that the rollout is stable. The rollout itself is never edited; the repair is limited to one guarded projection-cursor update.

I tested a real recovery with codex-cli 0.151.0 on Linux. The tool deliberately refuses malformed rollouts, unknown duplicate types, schema drift, active target sessions, and ambiguous cases.

This is an unofficial workaround for already-affected local sessions, not a fix for the upstream decoder or writer bug.

Preserve resume ordinals for float-bearing rollout records

Recover history projection past duplicate no-op ordinals

Preserve resume ordinals for float-bearing rollout records

Recover history projection past duplicate no-op ordinals

This is partially fixed in the upcoming 0.154.0 release (and the current alpha) by PR #42378. It forces decoding to go through canonical decoders, so it doesn't fail on the float. That was my failure mode that required database repair.

However, it doesn't fix the "reuses ordinals" half of the issue:

RolloutItem (codex-rs/history/src/lib.rs:102) has no catch-all variant, so an unrecognized type — a rollout written by a newer Codex, resumed by an older one — still makes decode_rollout_line fail. Upstream then walks past that record and the recorder appends at an ordinal already on disk.

Adding an independently tested, model-free reproduction and current stable-release results (September 5, 2026).

I see that #42378 has merged into main. I have not tested an alpha/build containing that change, so this is confirmation of stable-release impact and a reusable regression probe—not a claim that #42378 is ineffective.

Tested results

Runtime	Final token-count with nested primary, `used_percent: 1.0`	Same fixture, `rate_limits: null`	Same fixture followed by `task_complete`
macOS arm64, desktop-bundled `codex-cli 0.153.1`	Duplicate ordinal; projection stalls	Pass	Pass
Linux x86_64, standalone `codex-cli 0.153.3`	Duplicate ordinal; projection stalls	Pass	Pass
macOS arm64, official npm `codex-cli 0.153.4`, installed only in a temporary test directory	Duplicate ordinal; projection stalls	Pass	Pass

The four-record fixture ends at ordinal 3. After thread/resume and thread/settings/update, the runtime appends thread_settings_applied with ordinal 3 again; the projector reports:

failed to project durable rollout ... expected ordinal 4, got 3

Expected: append ordinal 4, then an EOF projection cursor expecting 5.

Additional macOS 0.153.1 minimization: preserving the rate-limit/credits objects but setting only primary: null also passes. Changing primary to a scalar is not a workaround; that variant still fails numbering.

This also occurred in real local and SSH-hosted histories: later completed turns remained in the canonical JSONL but disappeared from the reloaded UI because projection stopped at the duplicate. Backed-up, narrowly scoped cursor recovery restored the later turns without editing the transcripts, but that is recovery rather than prevention.

Standalone reproduction

The Python 3.10+ script below creates fresh temporary fixture directories and gives each child app-server its own CODEX_HOME. It copies no real histories, configuration, plugins, or credentials. It sends no turn/start and makes no model request. Synthetic fixture metadata uses cli_version: 0.153.1; the executable being tested is selected explicitly.

Run with an absolute path to the executable:

python3 reproduce_history_numbering.py --binary /absolute/path/to/codex --check

On the tested affected builds, the nested-primary case fails while both controls pass, and --check exits 1. Test artifacts are retained under the printed temporary directory. The script does not modify the production installation.

Complete tested reproduction script

"""Isolated Codex numbering regression probe. No model turns or real histories.

Creates synthetic fixtures only inside a fresh temporary Codex home. Native calls
are limited to initialize, settings/update, resume, and unsubscribe.
No credentials, user transcript contents, plugins, or user config are copied.
"""
import argparse
import asyncio
import datetime
import json
import os
import sqlite3
import tempfile
import uuid
from pathlib import Path


class Server:
    def __init__(self, binary, root):
        self.binary, self.root, self.counter = binary, root, 0

    async def __aenter__(self):
        # CODEX_HOME is scoped to this child process for its intended config-root
        # purpose; the user's environment and real home are never changed.
        env = dict(os.environ)
        env['CODEX_HOME'] = str(self.root)
        env['RUST_LOG'] = 'warn'
        for key in ('OPENAI_API_KEY', 'CODEX_API_KEY', 'CODEX_THREAD_ID'):
            env.pop(key, None)
        self.stderr_file = (self.root/'server.stderr.log').open('ab')
        self.proc = await asyncio.create_subprocess_exec(self.binary, 'app-server',
            '--listen', 'stdio://', cwd=str(self.root), env=env,
            stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
            stderr=self.stderr_file, limit=16*1024*1024)
        await self.request('initialize', {'clientInfo':{'name':'numbering_regression_probe','version':'1'},
                                          'capabilities':{'experimentalApi':True}})
        self.proc.stdin.write(b'{"method":"initialized"}\n')
        await self.proc.stdin.drain()
        return self

    async def request(self, method, params):
        allowed = {'initialize','thread/settings/update','thread/resume','thread/unsubscribe'}
        assert method in allowed
        self.counter += 1
        rid = self.counter
        self.proc.stdin.write((json.dumps({'id':rid,'method':method,'params':params})+'\n').encode())
        await self.proc.stdin.drain()
        while True:
            line = await asyncio.wait_for(self.proc.stdout.readline(),30)
            if not line:
                raise RuntimeError('Server exited before reply')
            obj = json.loads(line)
            if obj.get('id') == rid and ('result' in obj or 'error' in obj):
                if 'error' in obj:
                    raise RuntimeError(json.dumps(obj['error']))
                return obj['result']
            if 'id' in obj and 'method' in obj:
                raise RuntimeError('Unexpected server request: '+obj['method'])

    async def __aexit__(self, *args):
        self.proc.stdin.close()
        try:
            await asyncio.wait_for(self.proc.wait(),5)
        except asyncio.TimeoutError:
            self.proc.terminate()
            await asyncio.wait_for(self.proc.wait(),5)
        self.stderr_file.close()


def records(raw):
    with raw.open() as f:
        return [json.loads(line) for line in f]


async def case(binary, root, variant):
    root.mkdir(mode=0o700)
    tid = str(uuid.uuid4())
    session_dir=root/'sessions'/'2026'/'09'/'05'
    session_dir.mkdir(parents=True)
    raw=session_dir/('rollout-2026-09-05T07-00-00-'+tid+'.jsonl')
    ordinal = 0
    turn = str(uuid.uuid4())
    timestamp = datetime.datetime.now(datetime.timezone.utc).isoformat().replace('+00:00','Z')
    usage = dict(input_tokens=1,cached_input_tokens=0,cache_write_input_tokens=0,
                 output_tokens=1,reasoning_output_tokens=0,total_tokens=2)
    synthetic = [
        {'type':'session_meta','payload':{'id':tid,'session_id':tid,'timestamp':timestamp,
            'cwd':str(root),'source':'vscode','originator':'numbering regression probe',
            'cli_version':'0.153.1','model_provider':'openai','history_mode':'paginated',
            'base_instructions':{'text':'Synthetic test. Do not execute any model turn.'}}},
        {'type':'event_msg','payload':{'type':'task_started','turn_id':turn}},
        {'type':'response_item','payload':{'type':'message','role':'user','content':[
            {'type':'input_text','text':'Synthetic history numbering fixture. No model execution requested.'}]}},
        {'type':'event_msg','payload':{'type':'token_count','info':{
            'total_token_usage':usage,'last_token_usage':usage,'model_context_window':1000},
            'rate_limits':{'limit_id':'codex','limit_name':None,
                'primary':{'used_percent':1.0,'window_minutes':300,'resets_at':2000000000},
                'secondary':None,'credits':{'has_credits':False,'unlimited':False,'balance':'0'},
                'individual_limit':None,'spend_control_reached':None,'plan_type':'pro',
                'rate_limit_reached_type':None}}}
    ]
    if variant == 'no_rate_limits':
        synthetic[-1]['payload']['rate_limits'] = None
    elif variant == 'flat_primary':
        synthetic[-1]['payload']['rate_limits']['primary'] = 1.0
    elif variant == 'null_primary':
        synthetic[-1]['payload']['rate_limits']['primary'] = None
    elif variant == 'completed_tail':
        synthetic.append({'type':'event_msg','payload':{'type':'task_complete','turn_id':turn,
                                                       'last_agent_message':'Synthetic completion'}})
    # Write only the generated fixture, with no native process holding it open.
    with raw.open('a') as f:
        for item in synthetic:
            item.update(ordinal=ordinal,timestamp=timestamp)
            f.write(json.dumps(item,separators=(',',':'))+'\n')
            ordinal += 1
    before = records(raw)
    async with Server(binary,root) as server:
        await server.request('thread/resume',{'threadId':tid,'path':str(raw),'excludeTurns':True,
                                             'cwd':str(root),'model':'gpt-5.6-sol'})
        await server.request('thread/settings/update',{'threadId':tid,'effort':'medium'})
        await server.request('thread/unsubscribe',{'threadId':tid})
    after = records(raw)
    gaps = [{'previous':p['ordinal'],'current':c['ordinal'],
             'subtype':c.get('payload',{}).get('type')} for p,c in zip(after,after[1:])
            if c['ordinal'] != p['ordinal']+1]
    hist = sqlite3.connect('file:'+str(root/'thread_history_1.sqlite')+'?mode=ro')
    cursor = hist.execute('SELECT next_rollout_byte_offset,next_rollout_ordinal FROM thread_history_projection_state WHERE thread_id=?',(tid,)).fetchone()
    print(json.dumps({'variant':variant,'root':str(root),'thread_id':tid,'records_before':len(before),
        'records_after':len(after),'discontinuities':gaps,'cursor':cursor,'raw_size':raw.stat().st_size,
        'expected_next_ordinal':after[-1]['ordinal']+1}),flush=True)
    return not gaps and cursor == (raw.stat().st_size, after[-1]['ordinal']+1)


async def main():
    p=argparse.ArgumentParser()
    p.add_argument('--binary',default='/Applications/ChatGPT.app/Contents/Resources/codex')
    p.add_argument('--check',action='store_true',help='Exit nonzero if any numbering/projection invariant fails')
    p.add_argument('--variant',choices=['token_count_tail','no_rate_limits','completed_tail','flat_primary','null_primary'],action='append')
    a=p.parse_args()
    root=Path(tempfile.mkdtemp(prefix='codex-numbering-regression-')).resolve()
    print(json.dumps({'isolated_test_root':str(root),'binary':a.binary}),flush=True)
    results=[]
    for variant in a.variant or ['token_count_tail','no_rate_limits','completed_tail']:
        results.append(await case(a.binary,root/variant,variant))
    if a.check and not all(results):
        raise SystemExit(1)


if __name__ == '__main__':
    asyncio.run(main())

Would a stable backport of #42378 be possible? It would also help to document whether users with already-written duplicate ordinals need a separate supported projection-recovery step after upgrading.

Independent macOS arm64 confirmation on 0.153.4, with a seven-record numeric-encoding A/B reproduction

Confirmed using desktop-bundled codex-cli 0.153.4 on macOS arm64 (desktop version 26.901.41600). I am adding the current stable-version result and a small synthetic regression probe to this existing issue.

I see that #42378 has merged into main. I have not tested a build containing that change; the results below describe the installed 0.153.4 binary and do not claim the merged fix is ineffective.

The script below starts from a fresh isolated CODEX_HOME, initializes the schema with the native app-server, creates seven synthetic rollout records, and runs native migrate-rollouts --apply to generate the paginated history and index. The last record is token_count with info: null and a nested primary rate-limit window. It then starts a fresh app-server, resumes the synthetic thread, changes effort twice, and archives/unarchives to flush the writer. It submits no prompt and starts no model turn.

The integer control changes only the final used_percent JSON encoding from 1.0 to 1 and adjusts the projection's EOF byte offset. This edit is made after migration, because the native migration serializes the number back to 1.0.

Last `used_percent` encoding	Expected appended ordinals	Observed appended ordinals	Projection covers final EOF
`1.0`	7, 8	6, 7	No; next ordinal stays 7 while the file has 9 records
`1`	7, 8	7, 8	Yes; next ordinal is 9

The failing run logs:

thread history projection for 00000000-0000-7000-8000-000000000001 expected ordinal 7, got 6

In both controls, resume leaves the original file bytes unchanged and later writes preserve the original prefix. Neither run emits a turn/started notification. Thus this reproduction does not require an old user database, old-version session metadata, concurrent writers, or a real model response.

Expected: both equivalent numeric encodings must continue after the actual final persisted ordinal. Native-emitted telemetry must not cause subsequent history to become inaccessible. The result is consistent with the reverse-decoder problem described here; it is not evidence that the underlying persisted field should be an int32.

A related local investigation began because completed conversation turns disappeared after reopening despite intact canonical JSONL. No original transcripts, task identifiers, account details, or credentials are included below; the UUIDs, messages, and usage value are synthetic.

Standalone Python reproduction (float and integer controls)

Save as minimal_repro.py in an empty directory, then run:

python3 minimal_repro.py --kind float
python3 minimal_repro.py --kind integer

Use --codex /path/to/codex if the binary is elsewhere. The script creates separate disposable directories beside itself and refuses to overwrite existing runs. It does not patch the installed executable or use the normal Codex database directory.

#!/usr/bin/env python3
"""Synthetic Codex rollover reproduction; never opens the user's Codex home.

Run: python3 minimal_repro.py --kind float
     python3 minimal_repro.py --kind integer

Requires the installed Codex CLI. Each run creates its own disposable home,
native schema, synthetic history, and native index. It only resumes the synthetic
thread and changes settings. It never submits a prompt or starts a model turn.
"""
import argparse, hashlib, json, os, queue, sqlite3, subprocess, threading
from pathlib import Path

a=argparse.ArgumentParser()
a.add_argument('--kind',choices=['float','integer'],required=True)
a.add_argument('--codex',default='/Applications/ChatGPT.app/Contents/Resources/codex')
args=a.parse_args()
base=Path(__file__).resolve().parent/('minimal-'+args.kind)
if base.exists():raise SystemExit('Refusing to overwrite existing case directory: '+str(base))
lab=base/'home';lab.mkdir(parents=True);working=lab/'working';working.mkdir()
env=os.environ.copy();env['CODEX_HOME']=str(lab)
def cli(*arguments):
    r=subprocess.run([args.codex,*arguments],env=env,capture_output=True,text=True,timeout=45)
    if r.returncode:raise RuntimeError(r.stderr or r.stdout)
    return r.stdout
version=cli('--version').strip()
(lab/'config.toml').write_text('sandbox_mode = "read-only"\napproval_policy = "never"\n[analytics]\nenabled = false\n')
bootstrap=json.dumps({'id':1,'method':'initialize','params':{'clientInfo':{'name':'synthetic-schema-init','version':'1'}}})+'\n'
init=subprocess.run([args.codex,'app-server','--stdio'],env=env,input=bootstrap,capture_output=True,text=True,timeout=45)
(base/'schema-init.log').write_text(init.stderr)
if init.returncode:raise RuntimeError(init.stderr)
tid='00000000-0000-7000-8000-000000000001'
turn='00000000-0000-7000-8000-000000000002'
user='00000000-0000-7000-8000-000000000003'
agent='00000000-0000-7000-8000-000000000004'
time='2026-09-05T00:00:00.000Z';unix=1788566400
rollout=lab/'sessions/2026/09/05'/('rollout-2026-09-05T00-00-00-'+tid+'.jsonl')
rollout.parent.mkdir(parents=True)
def row(kind,payload):return {'timestamp':time,'type':kind,'payload':payload}
rows=[
 row('session_meta',{'session_id':tid,'id':tid,'timestamp':time,'cwd':str(working),'originator':'synthetic-repro',
     'cli_version':'0.153.4','source':'cli','thread_source':'user','model_provider':'openai',
     'base_instructions':{'text':''},'history_mode':'legacy'}),
 row('event_msg',{'type':'task_started','turn_id':turn,'started_at':unix,'model_context_window':1000,'collaboration_mode_kind':'default'}),
 row('response_item',{'type':'message','id':'synthetic_user','role':'user','content':[{'type':'input_text','text':'Synthetic test.'}]}),
 row('event_msg',{'type':'item_completed','thread_id':tid,'turn_id':turn,'item':{'type':'UserMessage','id':user,
     'client_id':None,'content':[{'type':'text','text':'Synthetic test.','text_elements':[]}]},'started_at_ms':unix*1000,'completed_at_ms':unix*1000}),
 row('response_item',{'type':'message','id':agent,'role':'assistant','content':[{'type':'output_text','text':'Synthetic response.'}],'phase':'commentary'}),
 row('event_msg',{'type':'item_completed','thread_id':tid,'turn_id':turn,'item':{'type':'AgentMessage','id':agent,
     'content':[{'type':'Text','text':'Synthetic response.'}],'phase':'commentary'},'started_at_ms':unix*1000,'completed_at_ms':unix*1000}),
 row('event_msg',{'type':'token_count','info':None,'rate_limits':{'primary':{'used_percent':1.0,'window_minutes':5,'resets_at':None}}})
]
rollout.write_text(''.join(json.dumps(r,separators=(',',':'))+'\n' for r in rows))
with sqlite3.connect(lab/'state_5.sqlite') as c:
    c.execute('INSERT INTO threads(id,rollout_path,created_at,updated_at,source,model_provider,cwd,title,sandbox_policy,approval_mode,cli_version,model,reasoning_effort,history_mode) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)',
              (tid,str(rollout),unix,unix,'cli','openai',str(working),'Synthetic reproduction','{"type":"read-only"}','never','0.153.4','gpt-6-astra','ultra','legacy'))
(base/'migration.json').write_text(cli('migrate-rollouts','--thread',tid,'--apply','--json'))
if args.kind=='integer':
    data=rollout.read_bytes();lines=data.splitlines(keepends=True)
    assert b'"used_percent":1.0' in lines[-1]
    lines[-1]=lines[-1].replace(b'"used_percent":1.0',b'"used_percent":1')
    data=b''.join(lines);rollout.write_bytes(data)
    with sqlite3.connect(lab/'thread_history_1.sqlite') as c:
        c.execute('UPDATE thread_history_projection_state SET next_rollout_byte_offset=? WHERE thread_id=?',(len(data),tid))
before=rollout.read_bytes();expected=len(before.splitlines())
err=(base/'app-server.stderr.log').open('w')
p=subprocess.Popen([args.codex,'app-server','--stdio'],env=env,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=err,text=True,bufsize=1)
q=queue.Queue();n=0;notifications=[]
def read_output():
    for line in p.stdout:q.put(json.loads(line))
threading.Thread(target=read_output,daemon=True).start()
def call(method,params):
    global n
    n+=1;p.stdin.write(json.dumps({'id':n,'method':method,'params':params})+'\n');p.stdin.flush()
    while True:
        r=q.get(timeout=45)
        if r.get('id')==n:
            if 'error' in r:raise RuntimeError(r)
            return r['result']
        notifications.append(r)
        if r.get('method')=='turn/started':raise RuntimeError('Unexpected model turn; stopping isolated process')
try:
    call('initialize',{'clientInfo':{'name':'synthetic-ordinal-repro','version':'1'},'capabilities':{'experimentalApi':True}})
    p.stdin.write('{"method":"initialized"}\n');p.stdin.flush()
    call('thread/resume',{'threadId':tid,'excludeTurns':True,'cwd':str(working),'runtimeWorkspaceRoots':[str(working)],
                         'sandbox':'read-only','approvalPolicy':'never','config':{'model_reasoning_effort':'ultra'}})
    after_resume=rollout.read_bytes()
    call('thread/settings/update',{'threadId':tid,'effort':'high'})
    call('thread/settings/update',{'threadId':tid,'effort':'ultra'})
    call('thread/archive',{'threadId':tid})
    call('thread/unarchive',{'threadId':tid})
finally:
    p.stdin.close()
    try:p.wait(timeout=10)
    except subprocess.TimeoutExpired:p.terminate();p.wait(timeout=5)
    err.close()
after=rollout.read_bytes();final_rows=[json.loads(l) for l in after.splitlines()]
with sqlite3.connect(lab/'thread_history_1.sqlite') as c:
    projection=c.execute('SELECT next_rollout_byte_offset,next_rollout_ordinal FROM thread_history_projection_state WHERE thread_id=?',(tid,)).fetchone()
result={'version':version,'kind':args.kind,'expected_first_append_ordinal':expected,
        'actual_appended_ordinals':[r['ordinal'] for r in final_rows[expected:]],
        'prefix_preserved':after.startswith(before),'resume_raw_unchanged':after_resume==before,
        'projection':projection,'eof':len(after),'line_count':len(final_rows),
        'projection_covers_eof':projection==(len(after),len(final_rows)),
        'model_turn_started':any(n.get('method')=='turn/started' for n in notifications)}
(base/'result.json').write_text(json.dumps(result,indent=2));print(json.dumps(result,indent=2))

Please consider including this numeric-encoding A/B probe in stable-release/backport validation, and documenting a supported recovery path for histories that already contain duplicated ordinals.

@silas-yangxu

@silas-yangxu @Alcray I have a small fork of Codex at v0.153.4 stable (changes vs release v0.153.4) which includes:

recovery path commit (this will let you resume broken sessions without the conversation loss)
hot-fix for this issue

If you'd like only the recovery and fixes, you can cherry-pick both of these two commits from my Codex fork on top of rust-v0.153.4 tag and build locally. I confirmed they cherry-pick cleanly.

IMO chances of backport are extremely small, and no automatic recovery has been implemented as far as I can tell. So if you got bit by this, you can use my patches, or you have to ask Codex to fix your SQLite database. Then hope to not encounter this bug again until 0.154.0 is released.

Independent Windows confirmation on 0.153.4, with a three-record, model-free reproduction

Adding a Windows confirmation and a small runtime probe to this existing issue. I have checked that #42378 is merged into main; I have not tested a build containing that change, so this reports the installed codex-cli 0.153.4 behavior, not a failure of the merged fix.

Controlled runtime results

Tested on Windows using the installed native codex-cli 0.153.4. The public script below was run again on September 6, 2026 with the same results:

Final record before resume	Expected ordinals after one settings update	Observed ordinals	Projection error
`token_count`, `used_percent: 26`	`[0,1,2,3]`	`[0,1,2,3]`	None
Same fixture, `used_percent: 26.0`	`[0,1,2,3]`	`[0,1,2,2]`	`expected ordinal 3, got 2`
Same fixture, `used_percent: 41.0`	`[0,1,2,3]`	`[0,1,2,2]`	`expected ordinal 3, got 2`
`26.0` token event followed by an ordinary `agent_message`	`[0,1,2,3,4]`	`[0,1,2,3,4]`	None

All initialize, resume, and settings-update calls succeeded. The appended record is event_msg/thread_settings_applied. The float cases also log the mismatch during shutdown.

The first three fixtures contain only session_meta(0), a synthetic agent_message(1), and token_count(2). Each case uses a fresh isolated CODEX_HOME and process. The only RPCs are initialize, thread/resume, and thread/settings/update; closing stdin flushes the recorder. No turn/start, model generation, forced crash, original conversation, or copied credentials are needed. This isolates terminal numeric encoding as a sufficient trigger; a crash is not necessary to reproduce it.

Observed user impact and recovery

In the affected desktop conversation, the canonical JSONL remained readable but had two resume-boundary ordinal duplicates, each immediately after a decimal-valued token_count. The history projection was stuck at the first duplicate and the UI showed only an old prefix. Restarting before repair did not recover it. This matches the desktop symptom described in #41566.

After backing up the data and performing a narrowly scoped local repair of those two internal records and the projection checkpoint, indexing resumed. All 94 conversation-message records present at the repair snapshot remained byte-for-byte identical. A subsequent app restart restored the visible conversation, confirmed by the user. This is recovery evidence, not a general-purpose repair recipe or a claim that the runtime bug is fixed.

Source correspondence

The 0.153.4 release source at commit 3d2ee51ca2d5db578f328aa75e20aa22c0197c9a still calls scanner.scan_next::<RolloutLine>() and skips rejected records. The existing canonical decoder documents the decimal/flattening issue. This agrees with the diagnosis already in this issue and the change in #42378.

Runnable reproduction

Save as codex-ordinal-repro.py and run with Python 3, passing the absolute path to the Codex executable to test:

python codex-ordinal-repro.py "<absolute-path-to-codex-executable>"

It creates synthetic cases in a new temporary directory and leaves their results there. The fixture's metadata/plan fields are synthetic constants, not account information. The script is a reproduction probe: its reported boolean checks describe the affected behavior and are not pass criteria for a fixed build.

Python script (standard library only; tested on Windows)

import json
import os
import queue
import subprocess
import threading
import time
import uuid
import sys
import tempfile
from pathlib import Path

EXE = Path(sys.argv[1]).resolve()
ROOT = Path(tempfile.mkdtemp(prefix='codex-ordinal-repro-'))

def run_case(name, percent, safe_tail=False):
    root = ROOT / name
    root.mkdir()
    home = root / 'isolated-home'
    home.mkdir()
    workspace = root / 'workspace'
    workspace.mkdir()
    (home / 'config.toml').write_text('[analytics]\nenabled = false\n', encoding='utf-8')
    tid = str(uuid.uuid4())
    session_dir = home / 'sessions' / '2026' / '09' / '06'
    session_dir.mkdir(parents=True)
    rollout = session_dir / ('rollout-2026-09-06T00-00-00-' + tid + '.jsonl')
    stamp = '2026-09-06T00:00:00Z'
    rows = [
        {'timestamp': stamp, 'ordinal': 0, 'type': 'session_meta', 'payload': {
            'id': tid, 'session_id': tid, 'timestamp': stamp, 'cwd': str(workspace),
            'originator': 'ordinal-diagnostic', 'cli_version': '0.153.3', 'source': 'exec',
            'model_provider': 'openai', 'history_mode': 'paginated',
            'base_instructions': {'text': 'Synthetic local history diagnostic.'}}},
        {'timestamp': stamp, 'ordinal': 1, 'type': 'event_msg', 'payload': {
            'type': 'agent_message', 'message': 'Synthetic record.'}},
        {'timestamp': stamp, 'ordinal': 2, 'type': 'event_msg', 'payload': {
            'type': 'token_count', 'info': None, 'rate_limits': {
                'limit_id': 'codex', 'limit_name': None,
                'primary': {'used_percent': percent, 'window_minutes': 10080, 'resets_at': 1800000000},
                'secondary': None, 'credits': None, 'individual_limit': None,
                'spend_control_reached': None, 'plan_type': 'prolite', 'rate_limit_reached_type': None}}},
    ]
    if safe_tail:
        rows.append({'timestamp': stamp, 'ordinal': 3, 'type': 'event_msg',
            'payload': {'type': 'agent_message', 'message': 'Synthetic trailing record.'}})
    rollout.write_text(''.join(json.dumps(r) + '\n' for r in rows), encoding='utf-8')
    env = os.environ.copy()
    for key in list(env):
        if any(s in key.upper() for s in ('API_KEY', 'AUTH_TOKEN', 'ACCESS_TOKEN')):
            del env[key]
    # This child uses a separate Codex configuration/data directory. No live thread is opened.
    env['CODEX_HOME'] = str(home)
    env['HTTP_PROXY'] = env['HTTPS_PROXY'] = env['ALL_PROXY'] = 'http://127.0.0.1:9'
    env['NO_PROXY'] = ''
    env['RUST_LOG'] = 'warn'
    output = queue.Queue()
    errors = []
    proc = subprocess.Popen([str(EXE), 'app-server', '--stdio'], cwd=workspace, env=env,
        stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        text=True, encoding='utf-8', creationflags=getattr(subprocess, 'CREATE_NO_WINDOW', 0))
    def reader():
        for line in proc.stdout:
            try: output.put(json.loads(line))
            except ValueError: pass
    def err_reader():
        for line in proc.stderr: errors.append(line)
    threading.Thread(target=reader, daemon=True).start()
    threading.Thread(target=err_reader, daemon=True).start()
    def request(ident, method, params, timeout=25):
        proc.stdin.write(json.dumps({'id': ident, 'method': method, 'params': params}) + '\n')
        proc.stdin.flush()
        end = time.monotonic() + timeout
        while time.monotonic() < end:
            try: msg = output.get(timeout=min(1, max(.01, end-time.monotonic())))
            except queue.Empty: continue
            if msg.get('id') == ident: return msg
        return {'timeout': method}
    try:
        init = request(1, 'initialize', {'clientInfo': {'name': 'ordinal_diagnostic', 'version': '1.0'},
            'capabilities': {'experimentalApi': True}})
        result = request(2, 'thread/resume', {'threadId': tid, 'path': str(rollout),
            'cwd': str(workspace), 'approvalPolicy': 'never', 'sandbox': 'read-only', 'model': 'gpt-6-astra'})
        update = request(3, 'thread/settings/update', {'threadId': tid, 'effort': 'low'})
        # Closing this isolated app-server flushes its synthetic recorder; no turn/start or model call is sent.
    finally:
        proc.stdin.close()
        try: proc.wait(timeout=8)
        except subprocess.TimeoutExpired:
            proc.terminate()
            proc.wait(timeout=5)
    after = [json.loads(line) for line in rollout.read_text(encoding='utf-8').splitlines()]
    data = {'case': name, 'percent_json': json.dumps(percent),
        'initialize_error': init.get('error'), 'resume_error': result.get('error'),
        'settings_error': update.get('error'),
        'timeout': result.get('timeout'), 'records_before': len(rows),
        'ordinals_after': [r.get('ordinal') for r in after],
        'types_after': [r.get('payload', {}).get('type', r.get('type')) for r in after],
        'projection_errors': [s.strip()[-500:] for s in errors if 'expected ordinal' in s],
        'isolated_directory': str(root)}
    (root/'result.json').write_text(json.dumps(data, indent=2), encoding='utf-8')
    return data

results = []
for name, value, tail in [('integer', 26, False), ('float', 26.0, False),
        ('float_41', 41.0, False), ('float_with_safe_tail', 26.0, True)]:
    result = run_case(name, value, tail)
    results.append(result)
    print(json.dumps(result), flush=True)
(ROOT/'results.json').write_text(json.dumps(results, indent=2), encoding='utf-8')
version = subprocess.check_output([str(EXE), '--version'], text=True).strip()
summary = {'runtime_version': version, 'cases': results,
    'assertions': {
        'integer_tail_continues': results[0]['ordinals_after'] == [0,1,2,3],
        'float_tail_duplicates': results[1]['ordinals_after'] == [0,1,2,2],
        'other_float_tail_duplicates': results[2]['ordinals_after'] == [0,1,2,2],
        'float_before_valid_tail_continues': results[3]['ordinals_after'] == [0,1,2,3,4],
        'all_resume_and_settings_calls_succeeded': all(not r['resume_error'] and not r['settings_error'] and not r['timeout'] for r in results)
    }}
(ROOT/'summary.json').write_text(json.dumps(summary, indent=2), encoding='utf-8')
print(json.dumps({'version': version, 'assertions': summary['assertions']}), flush=True)

Please include this Windows three-record probe when validating the release/backport of the canonical-decoder fix, and provide a supported recovery path for already affected paginated histories. No private conversation logs, database files, account identifiers, or personal filesystem paths are attached.

macOS 0.153.4: two/three-record offline probes reproduce one- and two-ordinal regressions, including `0.0`

Adding results from the installed official desktop-bundled binary, tested on September 9, 2026:

Desktop: ChatGPT.app / com.openai.codex, version 26.901.51231, build 8109.
Bundled codex-cli / app-server: 0.153.4.
macOS 26.6.2 (25G83), arm64.

I checked that #42378 is merged into main. These tests exercise the installed 0.153.4 binary; I have not tested a build containing that fix. This is not a report that the merged fix fails.

Controlled results

Each case uses a fresh isolated Codex data directory, a newly generated synthetic thread ID, and a local loopback provider configuration. The initial file contains only session_meta followed by one or two token_count records. Their info is null and the primary rate-limit window contains used_percent, window_minutes, and resets_at.

The RPC sequence is initialize → initialized → thread/resume → thread/settings/update → thread/unsubscribe, followed by closing stdin and waiting for the process. No turn/start, inference, forced crash, copied credentials, or real conversation is needed.

Initial records	`used_percent` JSON encoding	Expected next ordinal	Actual next ordinal	Resulting ordinals
metadata + two token-count records	`0`	3	3	`[0,1,2,3]`
metadata + one token-count record	`0.0`	2	1	`[0,1,1]`
metadata + two token-count records	`0.0`	3	1	`[0,1,2,1]`

The appended record is event_msg/thread_settings_applied in all three cases. Thus a nonzero percentage is not required, and the regression depth follows the number of trailing decimal-bearing records.

User-visible impact and separately validated recovery

Two local conversations exhibited the corresponding one-record and two-record regressions. In the latest case, the history API returned only five older turns even though later user messages and completed replies remained in the durable file. The stalled boundary was token_count(522), token_count(523), then thread_settings_applied(522), with the projector reporting expected ordinal 524, got 522. The file contained valid JSON throughout.

A backed-up copy with only top-level ordinals corrected was first verified in an isolated app-server, then applied after releasing the original writer. Native history projection recovered 13 turns and 618 items, and the history API returned the previously hidden final reply. All non-ordinal record content was compared for equality. No live SQLite rows were manually edited. This was a guarded, case-specific recovery, not a general recommendation to renumber arbitrary rollouts.

Please clarify the first desktop app build/bundled runtime that includes #42378 and the supported recovery behavior for already-affected histories. A one-time transcript recovery does not remove the initiating bug from the installed runtime. Related maintainer acknowledgment: #43142.

Self-contained Python 3 reproduction using the installed macOS bundled binary

Save this script in a new empty directory and run python3 reproduce_tail_ordinal.py. It uses only the Python standard library. It creates three private test directories beside the script and retains their results. Rerun from another empty directory; existing test directories are intentionally not overwritten. If testing another app installation, adjust only BINARY.

"""Offline black-box regression probe using synthetic history only.

Each child process uses a private Codex data directory. No live history,
credentials, or model inference is used. The configured provider is loopback.
"""
import asyncio
import json
import os
from pathlib import Path
import uuid

BINARY = "/Applications/ChatGPT.app/Contents/Resources/codex"
BASE = Path(__file__).resolve().parent


async def probe(label, usage_percent, tail_count):
    probe_dir = BASE / ("probe-" + label)
    probe_dir.mkdir(mode=0o700, exist_ok=False)
    workspace = probe_dir / "workspace"
    workspace.mkdir()
    data_dir = probe_dir / "codex-data"
    session_dir = data_dir / "sessions" / "2026" / "09" / "09"
    session_dir.mkdir(parents=True)
    (data_dir / "config.toml").write_text(
        'model = "offline-probe"\nmodel_provider = "offline-probe"\n'
        '[model_providers.offline-probe]\nname = "Offline probe"\n'
        'base_url = "http://127.0.0.1:9/v1"\nwire_api = "responses"\n'
        'requires_openai_auth = false\n[analytics]\nenabled = false\n'
    )
    thread_id = str(uuid.uuid4())
    rollout = session_dir / f"rollout-2026-09-09T00-00-00-{thread_id}.jsonl"
    timestamp = "2026-09-09T00:00:00Z"
    records = [{
        "timestamp": timestamp, "ordinal": 0, "type": "session_meta",
        "payload": {"id": thread_id, "timestamp": timestamp,
                    "cwd": str(workspace), "originator": "offline-history-probe",
                    "cli_version": "0.153.4", "source": "cli",
                    "model_provider": "offline-probe", "history_mode": "paginated"},
    }]
    for n in range(tail_count):
        records.append({
            "timestamp": timestamp, "ordinal": n + 1, "type": "event_msg",
            "payload": {"type": "token_count", "info": None,
                        "rate_limits": {"primary": {"used_percent": usage_percent,
                                                     "window_minutes": 5,
                                                     "resets_at": 1788912000}}},
        })
    rollout.write_text("".join(json.dumps(r) + "\n" for r in records))
    # CODEX_HOME retains its documented meaning, scoped only to this test child.
    # The user's shell environment and existing Codex data directory are untouched.
    child_env = {k: v for k, v in os.environ.items()
                 if not any(s in k.upper() for s in ("TOKEN", "SECRET", "API_KEY"))}
    child_env["CODEX_HOME"] = str(data_dir)
    stderr_file = (probe_dir / "stderr.log").open("wb")
    proc = await asyncio.create_subprocess_exec(
        BINARY, "app-server", "--stdio", cwd=workspace, env=child_env,
        stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
        stderr=stderr_file, limit=4 * 1024 * 1024,
    )
    request_id = 0
    async def rpc(method, params):
        nonlocal request_id
        request_id += 1
        proc.stdin.write((json.dumps({"id": request_id, "method": method,
                                     "params": params}) + "\n").encode())
        await proc.stdin.drain()
        while True:
            line = await asyncio.wait_for(proc.stdout.readline(), timeout=20)
            if not line:
                raise RuntimeError("test process exited before response")
            response = json.loads(line)
            if response.get("id") == request_id:
                if "error" in response:
                    raise RuntimeError(json.dumps(response["error"]))
                return response.get("result")
    result = {"case": label, "trailing_token_count": tail_count,
              "used_percent": usage_percent, "expected_next_ordinal": len(records)}
    try:
        await rpc("initialize", {"clientInfo": {"name": "offline-history-probe",
                  "version": "1"}, "capabilities": {"experimentalApi": True}})
        proc.stdin.write(b'{"method":"initialized"}\n')
        await proc.stdin.drain()
        await rpc("thread/resume", {"threadId": thread_id, "path": str(rollout),
                  "cwd": str(workspace), "model": "offline-probe",
                  "modelProvider": "offline-probe", "excludeTurns": True})
        await rpc("thread/settings/update", {"threadId": thread_id, "effort": "low"})
        await rpc("thread/unsubscribe", {"threadId": thread_id})
    except Exception as error:
        result["error"] = str(error)
    finally:
        proc.stdin.close()
        try:
            await asyncio.wait_for(proc.wait(), timeout=5)
        except asyncio.TimeoutError:
            proc.terminate()
            await asyncio.wait_for(proc.wait(), timeout=5)
        stderr_file.close()
    actual = [json.loads(line) for line in rollout.read_text().splitlines()]
    result["ordinals"] = [r.get("ordinal") for r in actual]
    result["appended_record_types"] = [r.get("payload", {}).get("type")
                                       for r in actual[len(records):]]
    result["actual_next_ordinal"] = (actual[len(records)].get("ordinal")
                                      if len(actual) > len(records) else None)
    result["reproduced"] = (result["actual_next_ordinal"] is not None and
                            result["actual_next_ordinal"] < len(records))
    (probe_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n")
    print(json.dumps(result), flush=True)
    return result


async def main():
    results = []
    for case in [("integer-control", 0, 2), ("one-float-tail", 0.0, 1),
                 ("two-float-tail", 0.0, 2)]:
        results.append(await probe(*case))
    (BASE / "reproduction-results.json").write_text(json.dumps(results, indent=2) + "\n")


if __name__ == "__main__":
    asyncio.run(main())

No original chat text, screenshots, real thread IDs, personal filesystem paths, credentials, databases, or raw production logs are attached.

The fix has been implemented since 0.154-alpha.1 (inclusive). Ask Codex to find an alpha build for the desktop app.

Fully fixed in 0.154.0 with recovery on resume.

Paginated history drops valid flattened rollout records and reuses ordinals #35746

Description

What version of Codex CLI is running?

What platform is your computer?

What issue are you seeing?

What steps can reproduce the bug?

What is the expected behavior?

Additional information

Activity

erichanwang commented on Jul 28, 2026

AlarmingMelon commented on Aug 2, 2026

AlarmingMelon commented on Aug 2, 2026

Additional affected task 1

Additional affected task 2

AlarmingMelon commented on Aug 3, 2026

Cross-device sequence

Read-only diagnostics on the Windows host

cis-aconiticacid commented on Aug 11, 2026

nos1609 commented on Aug 13, 2026

nos1609 commented on Aug 13, 2026

AaronZLT commented on Aug 14, 2026

nos1609 commented on Aug 15, 2026

liambern commented on Aug 15, 2026

Loong0x00 commented on Aug 15, 2026

Loong0x00 commented on Aug 15, 2026

17 remaining items

nos1609 commented on Aug 28, 2026

New 0.150.1 recurrence: task_started itself reuses the terminal token_count ordinal

Environment

Exact stalled boundary

Preserved but invisible answer

Impact

Fix direction

JavieHan commented on Aug 28, 2026

Independent macOS Desktop recurrence in two threads at the same recovery boundary

Environment

Two independent affected threads

Projection impact and bounded recovery

mulfyx commented on Aug 29, 2026

astral303 commented on Sep 3, 2026

Alcray commented on Sep 5, 2026

Tested results

Standalone reproduction

silas-yangxu commented on Sep 5, 2026

Independent macOS arm64 confirmation on 0.153.4, with a seven-record numeric-encoding A/B reproduction

astral303 commented on Sep 5, 2026

yotch0928-collab commented on Sep 6, 2026

Independent Windows confirmation on 0.153.4, with a three-record, model-free reproduction

Controlled runtime results

Observed user impact and recovery

Source correspondence

Runnable reproduction

xi-zhao commented on Sep 9, 2026

macOS 0.153.4: two/three-record offline probes reproduce one- and two-ordinal regressions, including 0.0

Controlled results

User-visible impact and separately validated recovery

astral303 commented on Sep 9, 2026

astral303 commented on Sep 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

New 0.150.1 recurrence: `task_started` itself reuses the terminal `token_count` ordinal

macOS 0.153.4: two/three-record offline probes reproduce one- and two-ordinal regressions, including `0.0`