Repository navigation
Rollout writer reuses the previous ordinal when thread_settings_applied follows a non-durable token_count, permanently stalling that thread's durable history projection #44609
Description
Summary
When a thread_settings_applied event is appended to a rollout immediately after a
non-durable token_count event, the writer assigns it the same ordinal as the
token_count. The rollout file then contains a duplicate ordinal, and the durable
history projector — which enforces strict monotonic ordinals — rejects it with:
failed to project durable rollout for <thread_id>: thread-store internal error:
thread history projection for <thread_id> expected ordinal 492, got 491
The projection cursor never advances past that point again. Every subsequent append
re-throws the same error for the life of the thread (392 occurrences on one of mine).
That thread's durable history is frozen from that moment on, silently.
User-visible symptom
The live event stream is unaffected, so nothing looks wrong while you stay in the
conversation — new turns stream in normally. The damage only shows when something
re-reads stored history:
- Reopening the thread shows it truncated at the stall point.
- On the mobile app it is worse, because it has no live buffer: the conversation loads
only up to the stall, you can reply and watch that reply stream fine, and then it
vanishes again the moment you leave and come back.
There is no error surfaced in the UI. The messages look lost.
Environment
- Codex app
26.903.9818.0(MSIX,OpenAI.Codex_2p2nqsd0c76g0) codex-cli 0.153.4- ChatGPT Desktop
1.2026.190.0 - Windows 11
10.0.26200 - Also reproduced on threads created under
0.150.0-alpha.12.2and0.152.1, so this
is not new to0.153.4.
Evidence
Every corrupted rollout has exactly one duplicated ordinal, and in all five cases the
duplicate is a token_count followed by a thread_settings_applied reusing its ordinal:
line 492: {"timestamp":"2026-09-10T14:26:16.022Z","ordinal":491,"type":"event_msg",
"payload":{"type":"token_count", ...
line 493: {"timestamp":"2026-09-10T14:26:56.774Z","ordinal":491,"type":"event_msg",
"payload":{"type":"thread_settings_applied", ...
The delay between the two varies from 30 seconds to 8 hours, so this is a settings-apply
landing on an idle thread whose last recorded event was a token_count.
Controlled comparison inside a single rollout
That same file contains nine thread_settings_applied events. Eight follow a durable
event and receive a correct fresh ordinal. The one that follows a token_count
duplicates. Nothing else differs:
| line | ordinal | preceding event | result |
|---|---|---|---|
| 366 | 365 | task_complete (364) |
ok |
| 382 | 381 | turn_aborted (380) |
ok |
| 493 | 491 | token_count (491) |
duplicate |
| 534 | 532 | response item (531) | ok |
| 595 | 593 | task_complete (592) |
ok |
| 618 | 616 | task_complete (615) |
ok |
| 723 | 721 | task_complete (720) |
ok |
| 1017 | 1015 | response item (1014) | ok |
| 1302 | 1300 | task_complete (1299) |
ok |
This points at the next-ordinal value being derived from the last durable item while
token_count still consumes an ordinal in the rollout file — the same non-durable
token_count ordinal accounting already described in #40747.
Failing call path
persist_rollout_items{item_count=1}:append_items{item_count=1}:append_items{item_count=1}:
failed to project durable rollout for <thread_id>: thread-store internal error:
thread history projection for <thread_id> expected ordinal 492, got 491
Target: codex_thread_store::local::live_writer. The same error also fires on the
shutdown path (op.dispatch.shutdown → "failed to project durable rollout during
shutdown"), so a clean exit does not repair it either.
Blast radius
5 of 800 local threads over roughly twelve days, earliest 2026-08-30. Combined, about
52 MB / ~1,300 rollout lines of history became unreadable while appearing intact in the
live view. Two of the five were hit 85 ms apart, which suggests an app-level settings
apply fanning out across open threads rather than any per-thread user action.
Useful discriminator when triaging: a thread whose projection cursor is behind its
rollout file but has no duplicate ordinal is just normal live lag and catches up. The
duplicate ordinal is what makes it permanent.
Detection
For each row in thread_history_projection_state, compare next_rollout_byte_offset to
the real size of the matching sessions/**/rollout-*.jsonl. Any thread that is behind
and whose file contains a duplicate "ordinal":N is permanently stalled.
Local repair (works, in case it helps others)
The stored byte offset already points at the duplicate line; only the ordinal is one too
high. With the app fully exited and the DB backed up:
UPDATE thread_history_projection_state
SET next_rollout_ordinal = next_rollout_ordinal - 1
WHERE thread_id IN (...);On reopening the thread, the projector consumed the remaining 27.9 MB and caught up
completely — cursor 491 → 1346, durable items 171 → 505, zero bytes behind, no further
errors. Recovered items included the user messages that had gone missing, so nothing is
actually lost on disk; it is purely a projection-cursor problem.
Note the projector is lazy — it backfills on thread open or next append, not at startup.
Suggested directions
- Assign the rollout ordinal from the actual last written rollout ordinal rather than
the last durable item, so non-durable events liketoken_countcannot leave a gap
that a later append reuses. - Have the projector treat
expected N+1, got Non a known non-durable predecessor as
recoverable — skip or resync the cursor — instead of hard-failing forever. - Surface a projection stall in the UI. Silent, permanent truncation of stored history
that still looks healthy in the live view is the worst part of this; it is only
visible in logs, and only if you know the string to grep for.
Related
- ChatGPT Beta cannot resume paginated rollout when inherited prefix ends on a non-durable event #40747 — resume fails when the inherited-prefix ordinal points at a non-durable
token_count; same accounting area. - Open in app allows two app-server processes to write the same rollout, merging divergent histories after restart #33241 — divergent rollout histories from two writers.
- Lost part of thread history and all subagent thread history after Codex App restart. #16599 — lost thread history after app restart; possibly the same root cause seen from
the user side.
Activity
github-actions commentedon Sep 10, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
ItsRobDude commentedon Sep 10, 2026
Follow-up with a wider scan, across 423 local session files (six days plus archived
sessions). It sharpens the claim in the original report, and partly corrects it.
A non-durable
token_countpredecessor appears necessary but not sufficient:thread_settings_appliedeventstoken_countThe necessary half is now on much firmer ground — zero duplicates in 2,472 cases where
the preceding event was durable.
But only 4 of 13 fired, so some further condition gates it. My original framing, built
on a nine-row comparison inside a single file, implied the
token_countpredecessor wasitself the discriminator. It isn't, and a naive repro will pass roughly two times in
three.
Candidates for the missing condition, in case they help narrow it: the length of the
idle gap before the settings-apply (the four failures span 30 s to 8 h), or whatever
reloads/rederives the in-memory next-ordinal value in between — an app restart, a thread
being evicted and rehydrated, or a settings fan-out across threads that were idle at the
time. Two of my four failures landed 85 ms apart, which fits the fan-out shape.
The four duplicates found by this scan are exactly the four already-known corrupt
threads in that window; the fifth predates it. So nothing additional broke.
2psyvwk7kf-ux commentedon Sep 10, 2026
Confirmation on macOS: duplicate ordinal, missing messages after restart
I can confirm the same failure pattern on macOS, affecting a long-running local task with generated images.
Environment
com.openai.codex, displayed as ChatGPT.User-visible impact
A completed batch of 10 generated images and its messages disappeared from the displayed conversation. After asking where the conversation had gone, the user reported an application crash; after reopening, that question also appeared to be missing. The messages, completion response and all 10 image files were subsequently confirmed to exist on disk. The exact process-crash cause has not been established, so I am reporting it as an accompanying symptom rather than a proven consequence of the ordinal defect.
Confirmed local evidence
The JSONL rollout has exactly one adjacent ordinal discontinuity in the inspected snapshot:
The duplicate occurs in records dated September 5, 2026. Every line in the inspected snapshot parsed as valid JSON. Later user messages and the completion response are present well beyond the duplicate.
The app logs repeat the following error after separate restarts:
Logger target:
codex_thread_store::local::live_writer.Expected behavior
Persisted messages and completed image results should remain accessible after reopening a task. If the history cannot be reconstructed, the UI should surface an error and provide a supported recovery path instead of showing an apparently truncated conversation.
Please include macOS in the scope of this issue and provide a supported repair for already affected histories. No live database or original rollout was modified during diagnosis. A readable local export was made to preserve access to the affected messages and image files.
This report omits private conversation contents, image contents, local usernames, full paths and thread identifiers.
sevendream77777-svg commentedon Sep 10, 2026
Additional independent Windows reproduction (local paginated task, observed September 6–10, 2026):
No raw rollout, database, task ID, private path, credentials, or message body is attached. Please include this as another corroborating Windows case and provide a preservation-first repair/reindex path for already affected threads. Do not advise deleting or resetting thread_history_1.sqlite or state_5.sqlite.
l307865548-dotcom commentedon Sep 11, 2026
Title: Codex desktop conversation freezes at an old turn after duplicated rollout ordinal
Environment: macOS; ChatGPT desktop / Codex app 26.903.61454, build 8378.
Observed: A long-running local task reopens at the turn from 2026-09-10 13:40 Asia/Shanghai. Later conversations through 2026-09-11 remain in the raw local JSONL, but are absent from the UI. The supported read_thread interface also returns that old turn as inProgress, rather than the subsequent completed turns.
Direct evidence:
Expected: Reopening the task should reconstruct all saved user/assistant messages and completed turns, and new messages should remain visible.
Please investigate duplicate restart-boundary ordinals and provide a backup-safe, per-thread history projection repair. Original logs and transcripts are preserved locally. This report omits private conversation content, user paths and credentials. No database or original transcript has been edited.
bandusix commentedon Sep 11, 2026
Independent reproduction: Claude-imported thread on macOS
I can confirm the same failure pattern on an imported session, which may help narrow the writer-side trigger.
Environment
codex-cli: 0.153.4Observed behavior
The imported history and later Codex turns were initially visible. After restarting/reloading the desktop app, the task consistently reopened at an older boundary. Approximately 20 later turns still existed in the canonical rollout JSONL, but the paginated history APIs stopped before them. The import operation itself had reported success with no failed records.
At the import-to-native-continuation boundary, the only ordinal anomaly in a 10,471-record snapshot was:
Every inspected line remained valid JSON, and the later user/assistant messages were present on disk.
Controlled recovery experiment
With the desktop app fully quit, I:
+1;The complete history immediately reappeared:
No message, tool payload, or other content was changed or deleted. This is a controlled diagnostic result, not a general repair recommendation; modifying canonical rollouts is unsafe without a verified backup and per-file validation.
This strongly confirms that the duplicate ordinal was the direct projection/pagination blocker in this imported thread. The import-specific trigger may be an off-by-one allocation or competing state transition when a successfully imported session is handed off to the native continuation writer.
Expected
thread_settings_applied.I am not attaching the full rollout or logs because they contain private project data. I can provide a sanitized minimal fixture or targeted diagnostics if maintainers need them.
tengweid commentedon Sep 11, 2026
Windows verification: the fix in #42378 is effective for our reproduced case in official CLI 0.154.0, while the installed Desktop 26.903.9818.0 backend 0.153.4 still reproduces the failure.
We compared signed Windows binaries using isolated CODEX_HOME directories, a sanitized existing-session fixture, and a localhost Responses SSE fixture. Each turn completed with a synthetic assistant response; no production model calls or live history edits were involved.
An additional upgrade test used a copy of the already damaged synthetic 0.153.4 result, including its duplicate ordinal and stalled projection. After a 0.154.0 resume and another completed append, all four assistant replies were readable after closing and restarting the server. We did not renumber or remove the existing duplicate record.
This matches #42378: the old flattened-envelope decoder can reject nested decimal values, and the reverse scanner consequently restores the writer ordinal from an earlier record. The new canonical decoder includes regression coverage for resume after a token-count record with a decimal rate-limit value.
Please urgently ship/enable the corresponding fixed backend in the Windows Desktop update channel. This has caused apparent conversation loss and interrupted project work. Please also provide supported Desktop upgrade/recovery guidance. Our installed Desktop still reports backend 0.153.4; the successful isolated CLI tests do not establish that every Desktop installation or every corruption scenario is fixed. The separate settings/project-registry reset report must not be conflated with this confirmed ordinal defect.