You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
macOS desktop history freezes at older turns: projection expected ordinal 3185, got 3184; migration says already_paginated #43124
A long-lived local conversation displays only older Thursday messages. Friday, Saturday and Sunday messages disappear from the desktop UI, while their actual user and assistant entries remain in the append-only local JSONL. The problem has recurred. This is not an archived thread.
Verified read-only evidence
Raw rollout is approximately 2.1 GiB, with over 24,000 JSONL records; a complete scan found zero malformed JSON lines.
Actual user/assistant message entries exist on September 4, 5 and 6, 2026 (UTC+08:00), not merely file modification timestamps.
The affected thread's thread_history_projection_state remains at next_rollout_byte_offset=53851170, next_rollout_ordinal=3185.
thread_items has 1,080 entries for this thread; max rollout ordinal is 3182 and latest created_at is 2026-09-03 09:27:56 UTC.
Desktop log records a failed thread/revert on 2026-09-05 12:48:55 UTC: failed to revert session: thread-store internal error: thread history projection ... expected ordinal 3185, got 3184
The desktop thread-reading interface similarly returns an old turn when asked for the newest turn, despite later durable messages.
The bundled CLI dry run codex migrate-rollouts --thread <affected-id> --json returns already_paginated, with 0 bytes processed; it does not offer a repair path.
Reproduction observed
Continue a local desktop conversation over several days, with images and tool output.
Reopen/view the task history.
Observe that later days of messages are absent from the UI despite remaining in the raw transcript.
Inspect projection state and logs: history projection is stuck with the ordinal mismatch above.
The exact initiating event is unknown; the failed revert is evidence of the inconsistency, not a proven cause.
Expected behavior
Saved conversation entries should remain accessible. A projection mismatch should recover safely from the intact durable transcript, or there should be a supported per-thread repair/reindex command. already_paginated must not prevent repair of an incomplete projection.
Related reports
Possibly the same issue family as #41986 and #42241, here reproduced on macOS and bundled CLI 0.153.4.
Privacy and safety
No conversation content, screenshots, credentials, raw session files, personal paths, or complete logs are attached. No SQLite rows or original transcript files have been edited or deleted during this investigation.
Thanks for the meticulous read-only evidence — this is exactly the kind of report that makes root-causing possible. I traced the ordinal-mismatch path against current main and can confirm the mechanism and the repair gap you flagged.
Where the exact error string comes from
expected ordinal 3185, got 3184 is emitted by codex-rs/thread-store/src/local/thread_history.rs:
// :144-149, inside apply_projectionif ordinal != next_ordinal {returnErr(ThreadStoreError::Internal{message:format!("thread history projection for {thread_id} expected ordinal {next_ordinal}, got {ordinal}"),});}
next_ordinal comes from the durable thread_history_projection_state.next_rollout_ordinal row (:120-132), and ordinal is the parsed ordinal of the next unprojected line in the rollout. So the failure means: the stored projection state claims the next ordinal is 3185, but the rollout line just past the stored byte checkpoint actually carries ordinal 3184.
Why current main can't produce this from a clean write
In the normal batch projector (thread_history_materialization.rs), a line whose ordinal < next_ordinal is skipped with a warning, not errored (:206-219), and the byte checkpoint still advances. So the ordinal-vs-byte checkpoint should stay consistent under normal appends.
The path that does raise this hard error is thread/revert: revert_thread.rs:73-93 calls materialize_to_sqlite on every lineage segment before computing the history base. When the projection state is already inconsistent (ordinal ahead of the actual line at the stored offset), that replay hits the :147 error and the whole revert fails — exactly the failed to revert session: ... expected ordinal 3185, got 3184 in your log. That's strong evidence the mismatch pre-existed and the failed revert is a symptom, not the cause (as you correctly suspected).
The repair gap (your main pain point)
Confirmed at rollout_migration.rs:513-517: when a thread's history_mode == Paginated, migrate-rollouts returns AlreadyPaginatedwithout checking whether the projection is complete or ordinal-consistent. So a transactionally half-advanced projection state (ordinal committed but rows not, or byte offset committed but ordinal off) has no repair path — --apply refuses to touch it, and no command re-projects from the intact transcript.
Suggested fix directions (server-side, either would address it)
Make migrate-rollouts --apply repair inconsistent paginated projections. When history_mode == Paginated but projection_state.next_rollout_byte_offset is not at the durable EOF (or ordinal continuity fails), treat it as repairable rather than already_paginated — re-open the rollout at the stored byte offset and re-run materialize_to_sqlite to reconcile ordinal vs byte checkpoint (the existing read_projection_steps already tolerates duplicate/regressed ordinals by skipping, so a re-replay converges).
Make thread/revert resilient. The :147 hard error during the pre-revert materialize_to_sqlite could be downgraded to a skippable anomaly (mirroring the ordinal < next_ordinal skip at :206), so a stale-but-intact projection doesn't brick an otherwise-valid revert.
Given the transcript is fully intact (zero malformed lines, 24k records) and the projection is the only thing off-by-one, fix #1 is the higher-value, lower-risk change — it recovers the visible history without touching the source data and closes the already_paginated repair dead-end.
Two questions that would help pin the divergence cause:
Do you have the next_rollout_byte_offset value from thread_history_projection_state (53851170) and can you confirm the rollout byte at that offset is the line carrying ordinal 3184 (not 3185)?
Was there any compaction / --id-preserve / concurrent-writer activity around 09-03, or is this purely a multi-day single-client desktop session?
(This looks like the same family as #41986 / #42157 / #42999 that the bot flagged — worth linking so the maintainers see the full repro range.)
I have the same issue. It happens in the VS Code extension for Linux (WSL2) and not macOS Desktop. There is an exact same apply_projection ordinal mismatch you traced that stalls a local thread here. I can also provide a possible case.
Environment:
VS Code extension Codex 26.901.22334, Remote-WSL
codex-cli 0.153.0
OS: Ubuntu on WSL2, kernel 6.6.87.2-microsoft-standard-WSL2, x86_64
thread is local, history_mode = paginated, long-lived, created in cli_version 0.151.0-alpha.7.2
Symptoms:
The UI transcript of the thread is frozen in an older turn. After this turn, all messages to the assistant and other users are recorded in the append only rollout, but are missing from the VS Code view.
codex resume <thread-id> of the bundled CLI also fails. Failed to resume session ...: failed to load a bounded thread item page. Therefore, this problem is in the thread-store/projection layer and is separate from the UI.
Likely trigger: a sudden resume
Both of the affected threads were interrupted in the middle of a turn (a reload of VS Code/sudden stop). The last assistant turn in each of these threads is incomplete. The most important clue is that both of these stuck projection cursors (next_rollout_byte_offset) point exactly to an event thread_settings_applied.
Thanks — this is the third independent platform report of the same stall (macOS desktop #43124, Windows #43142, and now Linux/WSL2 via the VS Code extension), and your thread_settings_applied clue is the missing piece that ties the writer-side stories together. A synthesis:
Why the stuck cursors point at thread_settings_applied
Your observation matches the other two reports exactly:
Yours (Linux/WSL2, 0.153.0): both stuck cursors point exactly at a thread_settings_applied event, trigger = interruption mid-turn then a fresh resume.
thread_settings_applied is not an ordinary turn-path event. It is emitted via send_event_raw_without_materializing_rollout (core/src/session/thread_settings.rs:111 → session/mod.rs:2360) — a raw path that appends straight into rollout storage without the normal turn sequencing. It is also the natural first append after a settings-restore during a fresh-process resume, which is why it is disproportionately the line the projector stalls on.
The unified shape of all three
A turn is interrupted mid-flight; the tail of the rollout ends in an incomplete turn whose last records frequently include event_msg/token_count (rate-limit tracking) — with a nested rate_limits block that, on the affected builds, carries a decimal float (e.g. used_percent: 82.0).
The fresh process resumes and computes the next ordinal with ordinal_state_for_rollout's backward scan. On 0.153.x that scan deserializes the flattened envelope directly, and nested decimals in the trailing token_count records make it bail past them → it derives a stale next ordinal (N−k, k = number of trailing decimal-bearing records).
The settings-restore append (thread_settings_applied) then writes with that stale ordinal — a duplicate/regressed top-level ordinal — and the projector cursor parks exactly there.
So "why thread_settings_applied?" = it is the first writer after a resume whose ordinal derivation is wrong, not because settings events are intrinsically special.
Why your codex resume also fails
That's a separate-but-related consequence: the resume path builds a bounded thread-item page from the projection; with the cursor parked at the regressed boundary the strict page load (apply_projection in thread_history.rs) hard-errors, so the CLI fails with failed to load a bounded thread item page even though the transcript is intact.
The fix status (important for your 0.153.0 build)
Both halves are fixed on current main but not in any 0.153.x release:
Route rollout reads through the canonical JSON decoder #42378 (canonical decoder, 69cebb5d15): the backward scan now decodes via decode_rollout_line, so trailing decimal token_count records no longer poison the next-ordinal derivation → no new reused ordinals on resume.
Verified by commit ancestry: stable rust-v0.153.4 contains neither; rust-v0.154.0-alpha.3 contains both. If you can update to a 0.154.0-line build and reopen the affected thread, the projection should advance past the thread_settings_applied boundary on its own — that verification (same task catches up + stays correct across a restart) is the open question on #43182 as well.
What's still missing upstream
Neither fix repairs an already-stalled task via a supported command, and the strict apply_projection error / already_paginated short-circuit in migrate-rollouts still has no recovery path (my earlier comment above). If you have the two next_rollout_byte_offset / next_rollout_ordinal values from thread_history_projection_state for your affected threads, that would confirm whether your boundary is a duplicate ordinal (like #43182) or a gap — it determines whether the skip logic in #42369 is sufficient or whether the strict path needs the repair direction from my first comment.
Thanks. I have now verified the boundary with a read-only inspection:
next_rollout_byte_offset = 53851170
next_rollout_ordinal = 3185
The surrounding records are:
Byte offset
Record
Top-level ordinal
53850290
event_msg/token_count, with decimal rate_limitsused_percent values of 10.0 and 62.0
3184
53851170
event_msg/thread_settings_applied
3184
53853160
event_msg/task_started
3185
So the checkpoint points exactly to a duplicate ordinal, not a forward gap.
Regarding compaction: yes, the rollout contains compacted records on September 3. The nearest ones before and after this boundary are:
2026-09-03T06:52:50.110Z, ordinal 2759
2026-09-03T09:34:07.155Z, ordinal 3307
The duplicate thread_settings_applied record is timestamped 2026-09-03T09:28:20.902Z. These records establish that compaction occurred, but do not establish it as the cause.
The session metadata identifies the originator as Codex Desktop and the creation-time CLI version as 0.152.1. I cannot currently confirm whether --id-preserve or concurrent writers were involved, so I do not want to rule those out without evidence.
The currently installed desktop app version is 26.901.51231, with bundled CLI 0.153.4. The projection remains stalled at the same checkpoint. Recovery using a build containing the fixes, including persistence across restart, has not yet been tested.
No original rollout or SQLite data has been modified during this investigation. Only structural metadata is included here; no raw chat content, personal paths, or complete logs are attached.
Thanks — this byte-level verification closes the loop on the mechanism, and your care with the investigation (read-only, metadata only) is appreciated.
Your boundary is a duplicate ordinal, not a forward gap — and that is exactly the shape the tolerant materializer already handles
Your table shows the checkpoint claims next_rollout_ordinal = 3185 while the line at the stored byte offset (53851170) carries 3184 — a repeated ordinal. On current main (thread_history_materialization.rs, the batch projector that runs on resume/open), a line whose ordinal < next_ordinal is skipped with a recorded DuplicateOrRegressedOrdinal anomaly, not errored, and the byte cursor advances past it. Reading forward from your checkpoint:
So the projection should self-heal on the next open/resume once you are on a build with #42369 (the tolerant materializer, in rust-v0.154.0-alpha.3 and later, not in any 0.153.x). #43262 is the empirical proof of this exact catch-up: a stalled 608MB/92k-record thread on Windows rebuilt its whole projection under 0.154.0-alpha.3 (bytes remaining 0, 27,857 items, 42 turns). #43182's reporter is currently verifying the same recovery on a second macOS instance with an identical thread_settings_applied duplicate boundary. Your verification criterion would be: reopen the thread on a 0.154+ build and confirm the frozen post-3307 turns become visible and stay visible across a restart.
Your ordinal sequence pins the writer-side mechanism precisely
Only the first append after the resume reused the stale ordinal; the very next writer derived 3185 correctly. That is the signature of the resume path computing its next ordinal from a backward scan that bails on the decimal-bearing token_count tail (pre-#42378 behavior — your thread was created on 0.152.1, which predates the canonical-decoder fix #42378), and thread_settings_applied being the first thing written after settings-restore on the fresh process. Once that first append landed, subsequent derivations saw the last durable ordinal (3184) and correctly continued at 3185.
Compaction timing
The duplicate sits at 09:28:20Z inside your compaction window (06:52Z → 09:34Z), which is suggestive that the interruption happened while compaction/settings activity was in flight — but I agree with your caution that this does not establish compaction as the cause, and the mechanism above does not require it: any mid-turn interruption followed by a fresh-process resume on a pre-#42378 build reproduces the stale-ordinal first append. If you want to confirm the trigger, a session log entry around 09:28:20Z showing an interruption/restart would settle it.
The only remaining gap upstream (unchanged)
No released or current-main command repairs an already-stalled projection in place on a pre-fix build: migrate-rollouts --apply short-circuits with already_paginated when history_mode == Paginated without checking projection consistency, and the strict apply_projection path (thread/revert) still hard-errors on the mismatch. The two fix directions from my earlier comment (repair-on-migrate, or downgrade the strict-path error to a skippable anomaly) remain open for maintainers.
Additional reproduction on macOS: the same projection failure occurs with a small, approximately 11 MB rollout, so a multi-gigabyte history is not required.
Environment: desktop bundle 26.901.51231 (8109), bundled CLI 0.153.4, macOS 27.0 (26A5378j), Apple Silicon. Bundle version read from Info.plist.
Observed behavior:
Recent messages repeatedly disappear from the existing conversation view.
The app's newest-first history interface returns only two September 5 turns, one still marked inProgress, and reports hasMore=false.
The same thread's canonical local JSONL contains later user messages through September 8 local time. A scan of 3,032 records found zero malformed JSON lines.
The persisted projection checkpoint is next_rollout_byte_offset=1540549 and next_rollout_ordinal=160.
The record at that exact byte offset is event_msg/thread_settings_applied with ordinal=159. The preceding record also has ordinal=159. This is the only ordinal discontinuity in that snapshot; later records are sequential.
Desktop logs repeatedly report: failed to project durable rollout for <thread>: thread-store internal error: thread history projection for <thread> expected ordinal 160, got 159.
Bundled codex migrate-rollouts --thread <thread> --json reports already_paginated and bytes_processed=0, leaving the projection stalled.
The ordinal mismatch is verified locally. Its initiating cause (restart, compaction, concurrent writer, etc.) has not been established. No source transcript or live database has been manually edited.
PR #42369 is merged, and GitHub's compare endpoint confirms rust-v0.154.0-alpha.3 contains that merge. At inspection time, GitHub's latest stable release is still rust-v0.153.4. The official prerelease archive was downloaded and its SHA-256 verified against the release asset digest, but history recovery with it has not been executed or validated here. The running desktop installation has not been replaced, and this report does not claim live recovery or prevention of recurrence.
Please ship the tolerant projection fix in the desktop runtime and provide a supported repair path for already-paginated histories. A successful history read should not silently present an old endpoint as complete when later durable entries cannot be projected.
No private conversation text, thread identifiers, personal paths, credentials, screenshots, or full logs are included.
Additional confirmed macOS case, with recurring failures across multiple local threads and a validated recovery of one thread.
Environment and impact
ChatGPT desktop 26.901.51231, build 8109.
Bundled Codex CLI 0.153.4; the separately installed CLI also reports 0.153.4.
Observed September 7, 2026; local paginated history.
Messages repeatedly disappear from the reconstructed conversation while later live messages still appear. This makes existing tasks effectively unusable and undermines confidence that work/conversation history is preserved.
A read-only audit of 591 directly mapped projection rows found seven additional stuck threads after repairing one. All seven checkpoints point to a record whose explicit ordinal repeats the preceding event_msg/token_count record's ordinal: two at task_started, five at thread_settings_applied. One failure occurs around ordinal 156, so very long conversations are not required. This is one user's local sample, not an estimate of prevalence across users. The other seven threads have not been modified.
Exact boundary in the repaired case
The true backend logger codex_thread_store::local::live_writer repeatedly emitted:
failed to project durable rollout for <thread>:
thread-store internal error: thread history projection for <thread>
expected ordinal 28739, got 28738
The source JSONL contains:
physical line 28739: event_msg/token_count, ordinal 28738
physical line 28740: event_msg/task_started, ordinal 28738
physical line 28741: next record, ordinal 28739
The index checkpoint was next_rollout_byte_offset=231733086, next_rollout_ordinal=28739. That byte offset points exactly to the duplicated task_started record. Every complete JSON record parsed successfully, and all subsequent explicit ordinals were sequential in this case. No indexed item creation/update or turn start/end occupied or exceeded the repeated position.
Observed recovery — not a general repair recommendation
After private backups and copy-only validation, we changed only this thread's next-ordinal projection cursor from 28739 to 28738, retaining the byte offset and the entire source transcript. The existing live writer immediately indexed 40 missing turns, advancing the item count from 8,567 to over 10,700. Subsequent writes continued advancing the index without the projection error. A native reader against an isolated snapshot returned the recovered turns.
The already-open desktop view remained stale after the index recovered. The user then restarted the app and confirmed the missing messages became visible.
This does not establish the original writer/resume trigger or prevent recurrence. Blindly subtracting one from other checkpoints, or skipping duplicate records containing meaningful turn boundaries, is not demonstrated safe. The five settings-applied variants were not assumed equivalent to the repaired task-start case.
Requested follow-up
Please ship prevention of ordinal reuse and a supported, content-preserving per-thread recovery path in the desktop runtime, with an actionable UI error rather than silently showing an old history endpoint. UI invalidation after recovery is also important.
Related to the boundary observations in #43142. We have not validated recovery using a newer/prerelease runtime and are not claiming a particular release fixes this case.
Privacy: no conversation text, task identifiers/titles, account identifiers, personal paths, credentials, screenshots, raw logs, or database/transcript files are attached.
Observed behavior
After reopening an existing local task, the desktop and its thread-reading API show only six older turns. Later user messages, completed assistant responses, tool execution, and externally verified GitHub changes remain in the original rollout. The last indexed turn is shown as interrupted, while several subsequent completed turns are absent.
Read-only evidence
Original rollout: 10,339,851 bytes, 1,026 JSONL records; every line parses successfully.
A later CLI resume attempt also logged an active-writer conflict; this is not established as the initiating cause.
Expected behavior / recovery request
Duplicate/regressed ordinals should not permanently hide later durable conversation history. Please provide or document a supported, per-thread way to rebuild an already-paginated projection from the intact rollout without modifying the original transcript or disturbing other running tasks.
No app databases or original rollout files were modified during diagnosis. The original transcript has been backed up locally. This comment intentionally omits conversation content, personal paths, thread IDs, credentials, and full logs.
The thread's stored cli_version is 0.153.4 (a thread-record value, not a verification of the currently running backend binary). Its history_mode is paginated.
Observed impact
Reopening a long-running conversation shows only four old turns, with the latest shown as interrupted. Several days of newer messages are absent from both the desktop UI and the app's thread-reading interface, although they remain in the durable rollout. The rollout ends with a final assistant response followed by task_complete.
Read-only evidence
Item
Observed value
Original rollout
147,325,561 bytes; 17,861 JSONL records
next_rollout_byte_offset
36,911,513
next_rollout_ordinal
3062
Record at the projection cursor
thread_settings_applied
Indexed history
4 turns; 778 items
Latest indexed item
2026-09-06 14:43:05 UTC
Durable final response and completion
2026-09-08 22:33:11 UTC
Later desktop resume
Succeeds, but still reports turnCount=4 and the old interrupted turn
The backend logs repeatedly report this sanitized error:
target: codex_thread_store::local::live_writer
failed to project durable rollout for <thread-id>:
thread-store internal error:
thread history projection for <thread-id>
expected ordinal 3062, got 3061
There are 334 matching entries in the retained logs. The mismatch also occurs during the final rollout flush and later shutdown. These are the earliest/latest retained matches, not proof of when the fault began.
Expected behavior
A durable-history projection failure should be visible and recoverable. The app should reconcile/rebuild the affected thread's history safely, or provide a supported per-thread repair path, instead of silently presenting an outdated conversation as the latest state.
The initiating event is unknown. No particular update, migration, restart, or concurrent-writer event has been established as the cause in this occurrence. No database or original transcript was modified during diagnosis, and no manual repair workaround was attempted.
Only sanitized diagnostic metadata is shared here. No server details, credentials, personal paths, project identifiers, thread IDs, prompts, conversation content, screenshots, or raw logs are included.
After reopening a long-lived local task, recent days of messages are absent. The app-provided newest-first history reader returns an old September 5 turn, while actual later user/assistant records remain in the durable JSONL, including September 9. A complete scan of the affected rollout found no malformed JSON records.
Sanitized diagnostic evidence:
Local history mode: paginated; affected rollout approximately 261 MB.
The logical thread has multiple lineage rollout segments; the stuck projection belongs to the latest segment rather than the root segment.
This confirms stale derived history rather than deletion of the observed recent messages. The original trigger for ordinal reuse has not been established. The database and raw transcripts were not modified during diagnosis. Recovery is being investigated separately; no successful repair is claimed here.
Please provide a supported per-thread repair path and identify the stable desktop/runtime release containing the applicable recovery handling. Related controlled-recovery report: #43742.
Privacy: no prompts, conversation content, screenshots, private paths, session identifiers, credentials, raw transcripts, or complete logs are attached.
Recovery follow-up
After preserving the transcript and using SQLite online backups, we tested the bundled 0.153.4 app-server in a fresh isolated directory without credentials or model turns. The snapshot's only ordinal discontinuity was token_count(17664), thread_settings_applied(17664), task_started(17665). The duplicate metadata occupied bytes 9573493 through 9577070.
Baseline native resume: latest segment remained at 3 turns / 91 items; full-lineage turns API returned 47 turns.
Guarded correction of this segment's next_rollout_byte_offset to 9577070, retaining next_rollout_ordinal=17665: native resume recovered 55 segment turns / 2823 items; full-lineage API returned 99 turns.
A fresh app-server process retained the recovered history and returned populated item pages, including a second page. All integrity checks returned ok. Snapshot bytes and SHA-256 were unchanged.
Negative tests rejected repeat execution, changed transcript prefix, a second duplicate, conflicting SQLite writer, and unexpected trigger writes; the trigger case rolled back the whole transaction.
The same one-row conditional UPDATE was then applied on the affected installation. The native writer advanced beyond 262 MB, current-day turns and user messages appeared in the derived store, and no new projection errors occurred in the immediate post-repair observation window. Original transcript prefix hash and inode were unchanged; SQLite integrity_check returned ok.
Desktop visual confirmation is still pending; API/database recovery is not claimed as a screenshot test. This is a narrowly verified recovery of one metadata boundary, not a general instruction to skip duplicate records or delete projection rows, nor a product-level fix for future ordinal reuse.
Additional verified macOS occurrence, diagnosed on 2026-09-10. This case has two repeated settings-event ordinals in one lineage segment, so repairing only the first boundary may leave a second stall later.
Environment
Desktop bundle: 26.903.71938 (build 8576), from installed Info.plist.
Bundled executable: codex-cli 0.153.4, verified with --version.
macOS 26.6.2, Apple Silicon / arm64.
Local long-running task, history_mode=paginated, with multiple lineage rollout segments.
User-visible failure
The user reports that completely quitting and reopening the app repeatedly restores the conversation to an older unfinished turn from September 9, 2026, approximately 15:53 UTC. Work and messages continued through September 10.
The app-provided newest-first read_thread interface independently returns that same old turn as inProgress, with no completion timestamp, while the task's updated timestamp is current. Later native user messages, assistant final answers, task-completion events, and the September 10 compaction summary are present in the durable JSONL. Project files also remain present. This establishes stale accessible history, not observed deletion of those messages.
Exact boundary evidence
The stalled projection belongs to the latest lineage segment, not the logical root task ID:
A sequential scan of 50,040,061 bytes / 3,430 JSONL records found zero JSON syntax failures and two top-level ordinal discontinuities. The second is another thread_settings_applied record at byte 32520584, timestamp 2026-09-10 06:31:26.449 UTC, repeating ordinal 12062. Its next record begins at byte 32524246. The scan's last ordinal was 13027. The file continues to grow; these are snapshot measurements, not fixed file dimensions. Syntactic validity does not establish semantic integrity of every event.
Sanitized error log excerpts
WARN codex_thread_store::local::live_writer
failed to project durable rollout for <logical-thread>:
thread-store internal error:
thread history projection for <lineage-segment>
expected ordinal 10022, got 10021
This signature appeared 329 times during live writes and 8 times during shutdown in one retained-log snapshot. Counts increased during diagnosis.
Desktop logs also contain:
2026-09-10T01:57:28.341Z
method=thread/revert
code=-32603
failed to revert session: thread-store internal error:
thread history projection for <lineage-segment>
expected ordinal 10022, got 10021
The same revert failure is present after later app restarts. This demonstrates that the revert path also encounters the boundary; it does not establish that revert caused the original duplicate.
Recovery inspection and requested outcome
The bundled command's default inspection mode was run without --apply:
It reports this for the affected segment and the other lineage segments, so this command did not rebuild the existing projection. Original transcripts and working files were backed up locally. No original transcript, projection checkpoint, app database, or cache was manually modified or deleted. The app was not force-terminated during diagnosis.
The 0.154.0 release changelog includes #42369 and #42378, and this issue has reports of recovery with newer runtimes. This occurrence has not been tested with 0.154.0, and no original-task recovery is claimed. Updating a standalone CLI would not itself establish that the desktop's bundled app-server changed.
Please identify the stable desktop/runtime release and supported per-thread recovery procedure for already-paginated affected tasks, including lineage segments and more than one duplicate boundary. Validation should recover the same original task's later messages and item pages, survive a fresh app restart, and surface an actionable error when projection cannot catch up.
Privacy: this comment contains only manually reviewed diagnostic metadata and shortened, redacted error excerpts. No conversation text, project data/source, private paths, task identifiers, credentials, screenshots, original transcripts, complete logs, or database files are attached. The first cause of ordinal reuse remains unestablished.
Confirmed: #42369 fully recovers both stalled threads on codex-cli 0.154.0-alpha.6.1.
Follow-up to my earlier report. Both threads had duplicate-ordinal boundaries: the stored next_rollout_byte_offset pointed at a thread_settings_applied whose ordinal repeated the previous line's. Both were created by a pre-#42378 build, the exact shape @argszero described:
After updating to a build bundling codex-cli 0.154.0-alpha.6.1 and reopening each thread, the tolerant materializer rebuilt the projection and the previously frozen turns are all visible again. Verified at the storage layer (thread_history SQLite), not just the UI:
Thread A - thread_items 383 > 10,051; projection cursor at end of rollout (396,499,967 / 396,499,967).
Thread B - thread_items 3,727 > 4,473 (max ordinal 12,911 > 15,283); projection cursor at end of rollout (63,012,985 / 63,012,985).
Because the cursor advanced to EOF and the items were written to the durable store, the recovery persists across a VS Code restart (it's not an in-memory-only rebuild). No new expected ordinal entries are logged after the rebuild.
So #42369 resolves the recovery path for duplicate-ordinal boundaries. The remaining gap you noted is unchanged. There's still no supported way to repair an already-stalled projection without moving to a fixed build: migrate-rollouts --apply short-circuits with already_paginated, and the strict apply_projection / thread/revert path still hard-errors on the mismatch.
Additional verified Windows occurrence, diagnosed on 2026-09-12. The user reports the same apparent history loss after reopening three times, seriously interrupting an ongoing work session. This comment adds a two-ordinal regression case and independently verified durable final-answer evidence.
Environment
Windows 11, build 26200, x64.
Installed MSIX: OpenAI.Codex 26.903.9818.0.
Actually executing backend binary: codex-cli 0.153.4, verified with --version.
Local thread uses history_mode=paginated.
Observed behavior
After reopening, recent completed messages are absent from the desktop history. The app-provided newest-first read_thread interface independently returns an older unfinished turn (status=inProgress, completedAt=null). It does not return the latest completed handoff.
The actual final assistant reply, matching the user's screenshot, remains in the raw JSONL at 2026-09-11 16:17:02.639 UTC, followed by task_complete at 16:17:03.121 UTC. The corresponding local handoff document also remains present. That completed turn has no projected assistant item.
Exact read-only evidence
At the 2026-09-12 00:44:50 UTC snapshot:
Raw rollout: 45,980,041 bytes; 3,821 complete JSONL records; zero JSON parse errors.
Projected history: 10 turns, latest still an old inProgress turn at ordinal 1828.
399 retained native live_writer warnings match the same ordinal mismatch (warning rows, not separate user incidents).
The mismatch persists during this diagnostic session.
Sanitized native error:
target: codex_thread_store::local::live_writer
failed to project durable rollout for <thread>:
thread-store internal error: thread history projection for <thread>
expected ordinal 1978, got 1976
Boundary:
Physical line
Byte offset
Record
Top-level ordinal
1977
17,473,323
event_msg/token_count
1976
1978
17,474,226
event_msg/token_count
1977
1979
17,475,129
event_msg/thread_settings_applied
1976
1980
17,477,161
event_msg/task_started
1977
1981
17,477,409
response_item/message
1978
The token_count records include decimal rate-limit values. The cursor points exactly to the regressed thread_settings_applied record. This establishes the failed projection boundary; it does not by itself establish the initiating cause.
PR #42369 and PR #42378 are merged and listed in the stable rust-v0.154.0 release changelog. The actual desktop backend here remains 0.153.4. Please confirm the supported desktop update/recovery path so existing affected histories are rebuilt and survive another complete restart. A standalone CLI update alone should not be represented as updating the desktop's actual backend.
Please also surface failed history projection to the user instead of showing a completed live reply that disappears from accessible history after restart.
Privacy and current status
A raw transcript snapshot and recovered user/final-answer transcript were preserved locally. No original transcript, live SQLite rows, application binaries, or configuration were edited. No business content, screenshots, personal paths, thread IDs, credentials, or raw logs are included in this public report. Desktop history recovery has NOT yet been verified.
Adding another verified macOS occurrence on September 12, 2026, with a successful local recovery and a small affected rollout (8.4 MB).
Environment and impact
Desktop app: 26.903.71938, build 8576.
Executing bundled backend: codex-cli 0.153.4, verified with the app's bundled binary.
macOS, Apple Silicon; affected thread uses history_mode=paginated.
Observed through a remote connection: the conversation stopped at an older unfinished turn, although newer requests had run and their completed responses were saved. The app-provided newest-first thread reader returned the same old ending.
Confirmed boundary and failure
The original rollout contained 3,027 valid JSONL records, 8,420,343 bytes, and 73 task_started events. Its SQLite history contained only 23 turns. There was exactly one top-level ordinal discontinuity:
Physical line (1-based)
Record
Ordinal
1504
event_msg/token_count
1503
1505
event_msg/thread_settings_applied
1503
1506
event_msg/task_started
1504
The projection cursor was stuck at next_rollout_byte_offset=5219151, next_rollout_ordinal=1504, precisely before the second record carrying ordinal 1503. Subsequent ordinals remained shifted by one.
Retained native logs contained 179 warnings with this signature, beginning at 2026-09-11 10:35:14 UTC and continuing during a newly completed request on September 12:
codex_thread_store::local::live_writer
failed to project durable rollout for <thread>:
thread-store internal error: thread history projection for <thread>
expected ordinal 1504, got 1503
The last projected turn was inProgress in SQLite but presented as interrupted by the reader. Later completed turns existed in the rollout. This misleading old status initially made the problem look like interrupted work.
A fresh temporary app-server process using the installed backend reproduced the same 23-turn history through both:
thread/turns/list, with both summary and full items;
thread/read, with includeTurns=true.
The migration dry run returned already_paginated, with zero bytes processed. This was therefore not just a stale UI window or remote transport cache.
Recovery verified
After testing on an isolated copy, a backed-up, offline repair corrected the ordinal sequence and rebuilt this thread's history projection. The final repaired rollout preserved every original record and its contents, changing only ordinal values; byte offsets were reconciled with that preserved file.
After applying the repair with the app/writer stopped:
Both history APIs returned all 73 turns, including the latest completed response and its artifact links.
Ordinals were contiguous and the projection reached EOF.
SQLite integrity_check passed.
The app reopened, and the user confirmed the missing messages were visible again.
This is evidence that the later conversation data was recoverable, not a recommendation for users to manually edit live chat storage. No newer backend was installed or tested in this recovery.
Remaining request
PRs #42369 and #42378 are merged, but the affected desktop installation was still running 0.153.4. Please identify the desktop release that bundles the relevant fixes and the supported recovery path for already-affected paginated threads. The UI should also clearly surface a stalled history projection instead of silently showing an older turn as the latest.
The duplicated ordinal and resulting projection failure are established. The initial event that caused ordinal reuse has not been established for this occurrence.
Only diagnostic metadata and a redacted error excerpt are included here; no conversation text, project names, user paths, thread identifiers, credentials, raw transcripts, or database files are attached.
Activity
github-actions commentedon Sep 5, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
argszero commentedon Sep 6, 2026
Thanks for the meticulous read-only evidence — this is exactly the kind of report that makes root-causing possible. I traced the ordinal-mismatch path against current
mainand can confirm the mechanism and the repair gap you flagged.Where the exact error string comes from
expected ordinal 3185, got 3184is emitted bycodex-rs/thread-store/src/local/thread_history.rs:next_ordinalcomes from the durablethread_history_projection_state.next_rollout_ordinalrow (:120-132), andordinalis the parsed ordinal of the next unprojected line in the rollout. So the failure means: the stored projection state claims the next ordinal is3185, but the rollout line just past the stored byte checkpoint actually carries ordinal3184.Why current
maincan't produce this from a clean writeIn the normal batch projector (
thread_history_materialization.rs), a line whoseordinal < next_ordinalis skipped with a warning, not errored (:206-219), and the byte checkpoint still advances. So the ordinal-vs-byte checkpoint should stay consistent under normal appends.The path that does raise this hard error is
thread/revert:revert_thread.rs:73-93callsmaterialize_to_sqliteon every lineage segment before computing the history base. When the projection state is already inconsistent (ordinal ahead of the actual line at the stored offset), that replay hits the:147error and the whole revert fails — exactly thefailed to revert session: ... expected ordinal 3185, got 3184in your log. That's strong evidence the mismatch pre-existed and the failed revert is a symptom, not the cause (as you correctly suspected).The repair gap (your main pain point)
Confirmed at
rollout_migration.rs:513-517: when a thread'shistory_mode == Paginated,migrate-rolloutsreturnsAlreadyPaginatedwithout checking whether the projection is complete or ordinal-consistent. So a transactionally half-advanced projection state (ordinal committed but rows not, or byte offset committed but ordinal off) has no repair path —--applyrefuses to touch it, and no command re-projects from the intact transcript.Suggested fix directions (server-side, either would address it)
Make
migrate-rollouts --applyrepair inconsistent paginated projections. Whenhistory_mode == Paginatedbutprojection_state.next_rollout_byte_offsetis not at the durable EOF (or ordinal continuity fails), treat it as repairable rather thanalready_paginated— re-open the rollout at the stored byte offset and re-runmaterialize_to_sqliteto reconcile ordinal vs byte checkpoint (the existingread_projection_stepsalready tolerates duplicate/regressed ordinals by skipping, so a re-replay converges).Make
thread/revertresilient. The:147hard error during the pre-revertmaterialize_to_sqlitecould be downgraded to a skippable anomaly (mirroring theordinal < next_ordinalskip at:206), so a stale-but-intact projection doesn't brick an otherwise-valid revert.Given the transcript is fully intact (zero malformed lines, 24k records) and the projection is the only thing off-by-one, fix #1 is the higher-value, lower-risk change — it recovers the visible history without touching the source data and closes the
already_paginatedrepair dead-end.Two questions that would help pin the divergence cause:
next_rollout_byte_offsetvalue fromthread_history_projection_state(53851170) and can you confirm the rollout byte at that offset is the line carrying ordinal 3184 (not 3185)?--id-preserve/ concurrent-writer activity around 09-03, or is this purely a multi-day single-client desktop session?(This looks like the same family as #41986 / #42157 / #42999 that the bot flagged — worth linking so the maintainers see the full repro range.)
marivaldojr commentedon Sep 6, 2026
Hi,
I have the same issue. It happens in the VS Code extension for Linux (WSL2) and not macOS Desktop. There is an exact same
apply_projectionordinal mismatch you traced that stalls a local thread here. I can also provide a possible case.Environment:
history_mode = paginated, long-lived, created incli_version 0.151.0-alpha.7.2Symptoms:
The UI transcript of the thread is frozen in an older turn. After this turn, all messages to the assistant and other users are recorded in the append only rollout, but are missing from the VS Code view.
codex resume <thread-id>of the bundled CLI also fails.Failed to resume session ...: failed to load a bounded thread item page. Therefore, this problem is in the thread-store/projection layer and is separate from the UI.Likely trigger: a sudden resume
Both of the affected threads were interrupted in the middle of a turn (a reload of VS Code/sudden stop). The last assistant turn in each of these threads is incomplete. The most important clue is that both of these stuck projection cursors (
next_rollout_byte_offset) point exactly to an eventthread_settings_applied.argszero commentedon Sep 6, 2026
Thanks — this is the third independent platform report of the same stall (macOS desktop #43124, Windows #43142, and now Linux/WSL2 via the VS Code extension), and your
thread_settings_appliedclue is the missing piece that ties the writer-side stories together. A synthesis:Why the stuck cursors point at
thread_settings_appliedYour observation matches the other two reports exactly:
thread_settings_appliedreusing ordinal 3499 (the persisted state expected 3500), immediately after atoken_countrecord.event_msg/token_countrecords followed by athread_settings_appliedreusing an earlier stored top-level ordinal.thread_settings_appliedevent, trigger = interruption mid-turn then a fresh resume.thread_settings_appliedis not an ordinary turn-path event. It is emitted viasend_event_raw_without_materializing_rollout(core/src/session/thread_settings.rs:111 → session/mod.rs:2360) — a raw path that appends straight into rollout storage without the normal turn sequencing. It is also the natural first append after a settings-restore during a fresh-process resume, which is why it is disproportionately the line the projector stalls on.The unified shape of all three
event_msg/token_count(rate-limit tracking) — with a nestedrate_limitsblock that, on the affected builds, carries a decimal float (e.g.used_percent: 82.0).ordinal_state_for_rollout's backward scan. On 0.153.x that scan deserializes the flattened envelope directly, and nested decimals in the trailingtoken_countrecords make it bail past them → it derives a stale next ordinal (N−k, k = number of trailing decimal-bearing records).thread_settings_applied) then writes with that stale ordinal — a duplicate/regressed top-level ordinal — and the projector cursor parks exactly there.So "why
thread_settings_applied?" = it is the first writer after a resume whose ordinal derivation is wrong, not because settings events are intrinsically special.Why your
codex resumealso failsThat's a separate-but-related consequence: the resume path builds a bounded thread-item page from the projection; with the cursor parked at the regressed boundary the strict page load (
apply_projectionin thread_history.rs) hard-errors, so the CLI fails withfailed to load a bounded thread item pageeven though the transcript is intact.The fix status (important for your 0.153.0 build)
Both halves are fixed on current
mainbut not in any 0.153.x release:69cebb5d15): the backward scan now decodes viadecode_rollout_line, so trailing decimaltoken_countrecords no longer poison the next-ordinal derivation → no new reused ordinals on resume.095ac4f131): lines withordinal < next_ordinalare now skipped with a recorded anomaly instead of erroring, so an existing parked cursor advances past the duplicate.Verified by commit ancestry: stable
rust-v0.153.4contains neither;rust-v0.154.0-alpha.3contains both. If you can update to a 0.154.0-line build and reopen the affected thread, the projection should advance past thethread_settings_appliedboundary on its own — that verification (same task catches up + stays correct across a restart) is the open question on #43182 as well.What's still missing upstream
Neither fix repairs an already-stalled task via a supported command, and the strict
apply_projectionerror /already_paginatedshort-circuit inmigrate-rolloutsstill has no recovery path (my earlier comment above). If you have the twonext_rollout_byte_offset/next_rollout_ordinalvalues fromthread_history_projection_statefor your affected threads, that would confirm whether your boundary is a duplicate ordinal (like #43182) or a gap — it determines whether the skip logic in #42369 is sufficient or whether the strict path needs the repair direction from my first comment.956005596 commentedon Sep 7, 2026
Thanks. I have now verified the boundary with a read-only inspection:
next_rollout_byte_offset = 53851170next_rollout_ordinal = 3185The surrounding records are:
53850290event_msg/token_count, with decimalrate_limitsused_percentvalues of10.0and62.0318453851170event_msg/thread_settings_applied318453853160event_msg/task_started3185So the checkpoint points exactly to a duplicate ordinal, not a forward gap.
Regarding compaction: yes, the rollout contains
compactedrecords on September 3. The nearest ones before and after this boundary are:2026-09-03T06:52:50.110Z, ordinal27592026-09-03T09:34:07.155Z, ordinal3307The duplicate
thread_settings_appliedrecord is timestamped2026-09-03T09:28:20.902Z. These records establish that compaction occurred, but do not establish it as the cause.The session metadata identifies the originator as
Codex Desktopand the creation-time CLI version as0.152.1. I cannot currently confirm whether--id-preserveor concurrent writers were involved, so I do not want to rule those out without evidence.The currently installed desktop app version is
26.901.51231, with bundled CLI0.153.4. The projection remains stalled at the same checkpoint. Recovery using a build containing the fixes, including persistence across restart, has not yet been tested.No original rollout or SQLite data has been modified during this investigation. Only structural metadata is included here; no raw chat content, personal paths, or complete logs are attached.
argszero commentedon Sep 7, 2026
Thanks — this byte-level verification closes the loop on the mechanism, and your care with the investigation (read-only, metadata only) is appreciated.
Your boundary is a duplicate ordinal, not a forward gap — and that is exactly the shape the tolerant materializer already handles
Your table shows the checkpoint claims
next_rollout_ordinal = 3185while the line at the stored byte offset (53851170) carries3184— a repeated ordinal. On currentmain(thread_history_materialization.rs, the batch projector that runs on resume/open), a line whoseordinal < next_ordinalis skipped with a recordedDuplicateOrRegressedOrdinalanomaly, not errored, and the byte cursor advances past it. Reading forward from your checkpoint:thread_settings_applied@ 3184 →3184 < 3185→ skip + advancetask_started@ 3185 → matchesnext_rollout_ordinal→ projection continues normallySo the projection should self-heal on the next open/resume once you are on a build with #42369 (the tolerant materializer, in
rust-v0.154.0-alpha.3and later, not in any 0.153.x). #43262 is the empirical proof of this exact catch-up: a stalled 608MB/92k-record thread on Windows rebuilt its whole projection under 0.154.0-alpha.3 (bytes remaining 0, 27,857 items, 42 turns). #43182's reporter is currently verifying the same recovery on a second macOS instance with an identicalthread_settings_appliedduplicate boundary. Your verification criterion would be: reopen the thread on a 0.154+ build and confirm the frozen post-3307 turns become visible and stay visible across a restart.Your ordinal sequence pins the writer-side mechanism precisely
token_count @ 3184 (decimal rate_limits)→thread_settings_applied @ 3184 (dup)→task_started @ 3185.Only the first append after the resume reused the stale ordinal; the very next writer derived 3185 correctly. That is the signature of the resume path computing its next ordinal from a backward scan that bails on the decimal-bearing
token_counttail (pre-#42378 behavior — your thread was created on 0.152.1, which predates the canonical-decoder fix #42378), andthread_settings_appliedbeing the first thing written after settings-restore on the fresh process. Once that first append landed, subsequent derivations saw the last durable ordinal (3184) and correctly continued at 3185.Compaction timing
The duplicate sits at 09:28:20Z inside your compaction window (06:52Z → 09:34Z), which is suggestive that the interruption happened while compaction/settings activity was in flight — but I agree with your caution that this does not establish compaction as the cause, and the mechanism above does not require it: any mid-turn interruption followed by a fresh-process resume on a pre-#42378 build reproduces the stale-ordinal first append. If you want to confirm the trigger, a session log entry around 09:28:20Z showing an interruption/restart would settle it.
The only remaining gap upstream (unchanged)
No released or current-main command repairs an already-stalled projection in place on a pre-fix build:
migrate-rollouts --applyshort-circuits withalready_paginatedwhenhistory_mode == Paginatedwithout checking projection consistency, and the strictapply_projectionpath (thread/revert) still hard-errors on the mismatch. The two fix directions from my earlier comment (repair-on-migrate, or downgrade the strict-path error to a skippable anomaly) remain open for maintainers.fangshengshi commentedon Sep 7, 2026
Additional reproduction on macOS: the same projection failure occurs with a small, approximately 11 MB rollout, so a multi-gigabyte history is not required.
Environment: desktop bundle 26.901.51231 (8109), bundled CLI 0.153.4, macOS 27.0 (26A5378j), Apple Silicon. Bundle version read from Info.plist.
Observed behavior:
failed to project durable rollout for <thread>: thread-store internal error: thread history projection for <thread> expected ordinal 160, got 159.codex migrate-rollouts --thread <thread> --jsonreports already_paginated and bytes_processed=0, leaving the projection stalled.The ordinal mismatch is verified locally. Its initiating cause (restart, compaction, concurrent writer, etc.) has not been established. No source transcript or live database has been manually edited.
PR #42369 is merged, and GitHub's compare endpoint confirms rust-v0.154.0-alpha.3 contains that merge. At inspection time, GitHub's latest stable release is still rust-v0.153.4. The official prerelease archive was downloaded and its SHA-256 verified against the release asset digest, but history recovery with it has not been executed or validated here. The running desktop installation has not been replaced, and this report does not claim live recovery or prevention of recurrence.
Please ship the tolerant projection fix in the desktop runtime and provide a supported repair path for already-paginated histories. A successful history read should not silently present an old endpoint as complete when later durable entries cannot be projected.
No private conversation text, thread identifiers, personal paths, credentials, screenshots, or full logs are included.
billsbooth commentedon Sep 7, 2026
Additional confirmed macOS case, with recurring failures across multiple local threads and a validated recovery of one thread.
Environment and impact
A read-only audit of 591 directly mapped projection rows found seven additional stuck threads after repairing one. All seven checkpoints point to a record whose explicit ordinal repeats the preceding
event_msg/token_countrecord's ordinal: two attask_started, five atthread_settings_applied. One failure occurs around ordinal 156, so very long conversations are not required. This is one user's local sample, not an estimate of prevalence across users. The other seven threads have not been modified.Exact boundary in the repaired case
The true backend logger
codex_thread_store::local::live_writerrepeatedly emitted:The source JSONL contains:
The index checkpoint was
next_rollout_byte_offset=231733086,next_rollout_ordinal=28739. That byte offset points exactly to the duplicatedtask_startedrecord. Every complete JSON record parsed successfully, and all subsequent explicit ordinals were sequential in this case. No indexed item creation/update or turn start/end occupied or exceeded the repeated position.Observed recovery — not a general repair recommendation
After private backups and copy-only validation, we changed only this thread's next-ordinal projection cursor from 28739 to 28738, retaining the byte offset and the entire source transcript. The existing live writer immediately indexed 40 missing turns, advancing the item count from 8,567 to over 10,700. Subsequent writes continued advancing the index without the projection error. A native reader against an isolated snapshot returned the recovered turns.
The already-open desktop view remained stale after the index recovered. The user then restarted the app and confirmed the missing messages became visible.
This does not establish the original writer/resume trigger or prevent recurrence. Blindly subtracting one from other checkpoints, or skipping duplicate records containing meaningful turn boundaries, is not demonstrated safe. The five settings-applied variants were not assumed equivalent to the repaired task-start case.
Requested follow-up
Please ship prevention of ordinal reuse and a supported, content-preserving per-thread recovery path in the desktop runtime, with an actionable UI error rather than silently showing an old history endpoint. UI invalidation after recovery is also important.
Related to the boundary observations in #43142. We have not validated recovery using a newer/prerelease runtime and are not claiming a particular release fixes this case.
Privacy: no conversation text, task identifiers/titles, account identifiers, personal paths, credentials, screenshots, raw logs, or database/transcript files are attached.
ZYC99 commentedon Sep 8, 2026
Additional independently verified reproduction on macOS with bundled
codex-cli 0.153.4. The symptoms andalready_paginateddead end match this report.Environment
26.901.51231(build8109).0.153.4.Darwin 25.6.0 arm64 arm.Observed behavior
After reopening an existing local task, the desktop and its thread-reading API show only six older turns. Later user messages, completed assistant responses, tool execution, and externally verified GitHub changes remain in the original rollout. The last indexed turn is shown as interrupted, while several subsequent completed turns are absent.
Read-only evidence
next_rollout_byte_offset=4869842,next_rollout_ordinal=339.codex_thread_store::local::live_writerwarnings:codex migrate-rollouts --thread <affected-thread> --jsonreportsalready_paginated,bytes_processed: 0.Expected behavior / recovery request
Duplicate/regressed ordinals should not permanently hide later durable conversation history. Please provide or document a supported, per-thread way to rebuild an already-paginated projection from the intact rollout without modifying the original transcript or disturbing other running tasks.
No app databases or original rollout files were modified during diagnosis. The original transcript has been backed up locally. This comment intentionally omits conversation content, personal paths, thread IDs, credentials, and full logs.
NODI-LAB commentedon Sep 9, 2026
Additional independently verified occurrence on Codex Desktop 26.901.51231 (build 8109), macOS 15.7.7 arm64.
The thread's stored
cli_versionis0.153.4(a thread-record value, not a verification of the currently running backend binary). Itshistory_modeispaginated.Observed impact
Reopening a long-running conversation shows only four old turns, with the latest shown as interrupted. Several days of newer messages are absent from both the desktop UI and the app's thread-reading interface, although they remain in the durable rollout. The rollout ends with a final assistant response followed by
task_complete.Read-only evidence
next_rollout_byte_offsetnext_rollout_ordinalthread_settings_appliedturnCount=4and the old interrupted turnThe backend logs repeatedly report this sanitized error:
There are 334 matching entries in the retained logs. The mismatch also occurs during the final rollout flush and later shutdown. These are the earliest/latest retained matches, not proof of when the fault began.
Expected behavior
A durable-history projection failure should be visible and recoverable. The app should reconcile/rebuild the affected thread's history safely, or provide a supported per-thread repair path, instead of silently presenting an outdated conversation as the latest state.
The initiating event is unknown. No particular update, migration, restart, or concurrent-writer event has been established as the cause in this occurrence. No database or original transcript was modified during diagnosis, and no manual repair workaround was attempted.
Only sanitized diagnostic metadata is shared here. No server details, credentials, personal paths, project identifiers, thread IDs, prompts, conversation content, screenshots, or raw logs are included.
somethingcubic commentedon Sep 9, 2026
Additional independently observed occurrence on macOS, desktop bundle 26.903.61454, bundled codex-cli 0.153.4.
After reopening a long-lived local task, recent days of messages are absent. The app-provided newest-first history reader returns an old September 5 turn, while actual later user/assistant records remain in the durable JSONL, including September 9. A complete scan of the affected rollout found no malformed JSON records.
Sanitized diagnostic evidence:
failed to project durable rollout for <thread>: thread history projection for <rollout> expected ordinal 17665, got 17664.codex migrate-rollouts --thread <redacted> --jsonreturns already_paginated, bytes_processed=0.This confirms stale derived history rather than deletion of the observed recent messages. The original trigger for ordinal reuse has not been established. The database and raw transcripts were not modified during diagnosis. Recovery is being investigated separately; no successful repair is claimed here.
Please provide a supported per-thread repair path and identify the stable desktop/runtime release containing the applicable recovery handling. Related controlled-recovery report: #43742.
Privacy: no prompts, conversation content, screenshots, private paths, session identifiers, credentials, raw transcripts, or complete logs are attached.
Recovery follow-up
After preserving the transcript and using SQLite online backups, we tested the bundled 0.153.4 app-server in a fresh isolated directory without credentials or model turns. The snapshot's only ordinal discontinuity was token_count(17664), thread_settings_applied(17664), task_started(17665). The duplicate metadata occupied bytes 9573493 through 9577070.
Desktop visual confirmation is still pending; API/database recovery is not claimed as a screenshot test. This is a narrowly verified recovery of one metadata boundary, not a general instruction to skip duplicate records or delete projection rows, nor a product-level fix for future ordinal reuse.
shunyi-wu commentedon Sep 10, 2026
Additional verified macOS occurrence, diagnosed on 2026-09-10. This case has two repeated settings-event ordinals in one lineage segment, so repairing only the first boundary may leave a second stall later.
Environment
--version.history_mode=paginated, with multiple lineage rollout segments.User-visible failure
The user reports that completely quitting and reopening the app repeatedly restores the conversation to an older unfinished turn from September 9, 2026, approximately 15:53 UTC. Work and messages continued through September 10.
The app-provided newest-first
read_threadinterface independently returns that same old turn asinProgress, with no completion timestamp, while the task's updated timestamp is current. Later native user messages, assistant final answers, task-completion events, and the September 10 compaction summary are present in the durable JSONL. Project files also remain present. This establishes stale accessible history, not observed deletion of those messages.Exact boundary evidence
The stalled projection belongs to the latest lineage segment, not the logical root task ID:
Records at and around that cursor:
A sequential scan of 50,040,061 bytes / 3,430 JSONL records found zero JSON syntax failures and two top-level ordinal discontinuities. The second is another
thread_settings_appliedrecord at byte 32520584, timestamp 2026-09-10 06:31:26.449 UTC, repeating ordinal 12062. Its next record begins at byte 32524246. The scan's last ordinal was 13027. The file continues to grow; these are snapshot measurements, not fixed file dimensions. Syntactic validity does not establish semantic integrity of every event.Sanitized error log excerpts
This signature appeared 329 times during live writes and 8 times during shutdown in one retained-log snapshot. Counts increased during diagnosis.
Desktop logs also contain:
The same revert failure is present after later app restarts. This demonstrates that the revert path also encounters the boundary; it does not establish that revert caused the original duplicate.
Recovery inspection and requested outcome
The bundled command's default inspection mode was run without
--apply:It reports this for the affected segment and the other lineage segments, so this command did not rebuild the existing projection. Original transcripts and working files were backed up locally. No original transcript, projection checkpoint, app database, or cache was manually modified or deleted. The app was not force-terminated during diagnosis.
The 0.154.0 release changelog includes #42369 and #42378, and this issue has reports of recovery with newer runtimes. This occurrence has not been tested with 0.154.0, and no original-task recovery is claimed. Updating a standalone CLI would not itself establish that the desktop's bundled app-server changed.
Please identify the stable desktop/runtime release and supported per-thread recovery procedure for already-paginated affected tasks, including lineage segments and more than one duplicate boundary. Validation should recover the same original task's later messages and item pages, survive a fresh app restart, and surface an actionable error when projection cannot catch up.
Privacy: this comment contains only manually reviewed diagnostic metadata and shortened, redacted error excerpts. No conversation text, project data/source, private paths, task identifiers, credentials, screenshots, original transcripts, complete logs, or database files are attached. The first cause of ordinal reuse remains unestablished.
marivaldojr commentedon Sep 11, 2026
Confirmed: #42369 fully recovers both stalled threads on
codex-cli 0.154.0-alpha.6.1.Follow-up to my earlier report. Both threads had duplicate-ordinal boundaries: the stored
next_rollout_byte_offsetpointed at athread_settings_appliedwhose ordinal repeated the previous line's. Both were created by a pre-#42378 build, the exact shape @argszero described:thread_settings_applied@ 12916 (dup) >task_started@ 12917After updating to a build bundling
codex-cli 0.154.0-alpha.6.1and reopening each thread, the tolerant materializer rebuilt the projection and the previously frozen turns are all visible again. Verified at the storage layer (thread_history SQLite), not just the UI:thread_items383 > 10,051; projection cursor at end of rollout (396,499,967 / 396,499,967).thread_items3,727 > 4,473 (max ordinal 12,911 > 15,283); projection cursor at end of rollout (63,012,985 / 63,012,985).Because the cursor advanced to EOF and the items were written to the durable store, the recovery persists across a VS Code restart (it's not an in-memory-only rebuild). No new expected ordinal entries are logged after the rebuild.
So #42369 resolves the recovery path for duplicate-ordinal boundaries. The remaining gap you noted is unchanged. There's still no supported way to repair an already-stalled projection without moving to a fixed build:
migrate-rollouts --applyshort-circuits withalready_paginated, and the strictapply_projection/thread/revertpath still hard-errors on the mismatch.bobbyleeweiwei-bobby commentedon Sep 12, 2026
Additional verified Windows occurrence, diagnosed on 2026-09-12. The user reports the same apparent history loss after reopening three times, seriously interrupting an ongoing work session. This comment adds a two-ordinal regression case and independently verified durable final-answer evidence.
Environment
Observed behavior
After reopening, recent completed messages are absent from the desktop history. The app-provided newest-first read_thread interface independently returns an older unfinished turn (status=inProgress, completedAt=null). It does not return the latest completed handoff.
The actual final assistant reply, matching the user's screenshot, remains in the raw JSONL at 2026-09-11 16:17:02.639 UTC, followed by task_complete at 16:17:03.121 UTC. The corresponding local handoff document also remains present. That completed turn has no projected assistant item.
Exact read-only evidence
At the 2026-09-12 00:44:50 UTC snapshot:
Sanitized native error:
Boundary:
The token_count records include decimal rate-limit values. The cursor points exactly to the regressed thread_settings_applied record. This establishes the failed projection boundary; it does not by itself establish the initiating cause.
The bundled read-only command:
does not repair it.
Recovery / release request
PR #42369 and PR #42378 are merged and listed in the stable rust-v0.154.0 release changelog. The actual desktop backend here remains 0.153.4. Please confirm the supported desktop update/recovery path so existing affected histories are rebuilt and survive another complete restart. A standalone CLI update alone should not be represented as updating the desktop's actual backend.
Please also surface failed history projection to the user instead of showing a completed live reply that disappears from accessible history after restart.
Privacy and current status
A raw transcript snapshot and recovered user/final-answer transcript were preserved locally. No original transcript, live SQLite rows, application binaries, or configuration were edited. No business content, screenshots, personal paths, thread IDs, credentials, or raw logs are included in this public report. Desktop history recovery has NOT yet been verified.
Related Windows reports: #42197 and #44432.
VincentMeijer commentedon Sep 12, 2026
Adding another verified macOS occurrence on September 12, 2026, with a successful local recovery and a small affected rollout (8.4 MB).
Environment and impact
history_mode=paginated.Confirmed boundary and failure
The original rollout contained 3,027 valid JSONL records, 8,420,343 bytes, and 73
task_startedevents. Its SQLite history contained only 23 turns. There was exactly one top-level ordinal discontinuity:event_msg/token_countevent_msg/thread_settings_appliedevent_msg/task_startedThe projection cursor was stuck at
next_rollout_byte_offset=5219151,next_rollout_ordinal=1504, precisely before the second record carrying ordinal 1503. Subsequent ordinals remained shifted by one.Retained native logs contained 179 warnings with this signature, beginning at 2026-09-11 10:35:14 UTC and continuing during a newly completed request on September 12:
The last projected turn was
inProgressin SQLite but presented as interrupted by the reader. Later completed turns existed in the rollout. This misleading old status initially made the problem look like interrupted work.A fresh temporary app-server process using the installed backend reproduced the same 23-turn history through both:
thread/turns/list, with both summary and full items;thread/read, withincludeTurns=true.The migration dry run returned
already_paginated, with zero bytes processed. This was therefore not just a stale UI window or remote transport cache.Recovery verified
After testing on an isolated copy, a backed-up, offline repair corrected the ordinal sequence and rebuilt this thread's history projection. The final repaired rollout preserved every original record and its contents, changing only ordinal values; byte offsets were reconciled with that preserved file.
After applying the repair with the app/writer stopped:
integrity_checkpassed.This is evidence that the later conversation data was recoverable, not a recommendation for users to manually edit live chat storage. No newer backend was installed or tested in this recovery.
Remaining request
PRs #42369 and #42378 are merged, but the affected desktop installation was still running 0.153.4. Please identify the desktop release that bundles the relevant fixes and the supported recovery path for already-affected paginated threads. The UI should also clearly surface a stalled history projection instead of silently showing an older turn as the latest.
The duplicated ordinal and resulting projection failure are established. The initial event that caused ordinal reuse has not been established for this occurrence.
Only diagnostic metadata and a redacted error excerpt are included here; no conversation text, project names, user paths, thread identifiers, credentials, raw transcripts, or database files are attached.