Repository navigation
Thread store resolves stale session_index entries to missing rollout files (state db returned stale rollout path) #31074
Description
Summary
The local session index (~/.codex/session_index.jsonl) retains entries whose rollout
JSONL no longer exists in either the live sessions tree or archived_sessions/. When the
app/thread store resolves such a thread id, it returns a path to a missing file and errors
instead of falling back to the archived copy or self-healing the index. On a
long-lived, high-session-volume host this accumulates: 113 of 283 indexed entries (40%)
had no backing rollout file anywhere.
Evidence
Codex's own log reports the inconsistency from inside codex_rollout::list:
ERROR codex_rollout::list: state db returned stale rollout path for thread
019f2339-9299-7e51-b192-9751a0a03586:
/home/USER/.codex/sessions/2026/07/02/rollout-2026-07-02T14-26-43-019f2339-...jsonl
That file is not in live sessions/, not in archived_sessions/ (507 archived JSONLs
present), and not found by a recursive basename search.
Audit of the index on the affected host:
index entries: 283 | present (live or archived): 170 | STALE (no backing file): 113
A second, related failure mode also seen: a rollout file that exists but is misfiled —
stored under sessions/2026/05/09/ while its contents begin with 2026-07-03 data and do
not start with a session_meta record. So the index/path bookkeeping and the on-disk
partitioning can disagree in both directions (missing file; misplaced/malformed file).
How it likely arises
The index write and the rollout file lifecycle (create / archive-move / delete) are not
crash-consistent. If the process is interrupted between updating the index and finalizing
the file — e.g. an ENOSPC disk-full (our trigger: root fs hit 100%), a SIGKILL, or an
archival move that half-completes — the index keeps an entry the filesystem can't satisfy.
Nothing later reconciles it.
Impact
- App/thread resolution errors when opening or listing affected threads.
- Users cannot cleanly resume or reference historical sessions that are actually still
recoverable from durable artifacts. - Grows unbounded on active hosts; 40% stale observed.
Expected behavior (any of)
- On resolve-miss, fall back to
archived_sessions/(and a basename search) before erroring. - Self-heal the index: drop or flag entries whose rollout file is absent from both live
and archived locations, rather than repeatedly resolving to a dead path. - Make the index update + rollout finalize crash-consistent (write file first / fsync +
atomic rename, update index only after), so an interrupted write can't orphan an entry. - Provide a
codexsubcommand to verify/repair the session index.
Workaround in use
Local reconciliation script: back up session_index.jsonl, drop entries with no backing
rollout file in live or archived sessions (never deletes rollout files). Mitigates the
resolve error until a first-class fix lands.
Related issues (adjacent but distinct)
Several open issues touch "stale rollout path," but each is tied to a specific Desktop UI
flow rather than the crash/interruption-consistency root cause and the systemic 40%
accumulation reported here (observed via codex-cli / app-server, not a UI action):
- Archiving active Desktop chat leaves stale active session and stale archived rollout path after unarchive #26201 / Codex Desktop unarchive leaves stale archived rollout path #26174 — stale archived rollout path after archive/unarchive
- Windows Desktop: failed to resolve rollout path / Error submitting message on every new conversation #30924 — "failed to resolve rollout path" on new conversations (Windows Desktop)
- Codex Desktop delete leaves local session index, logs, and desktop-state residues #23991 — delete leaves session-index residue
- Codex App on macOS crashes with SIGABRT after sending a message; earlier missing rollout path #21663 — missing rollout path → macOS SIGABRT
This report is about the index and rollout-file lifecycle becoming inconsistent when a
write is interrupted (ENOSPC / SIGKILL / half-completed archival move), independent of
any UI flow, and the index then never self-reconciling. Please link/dedupe as appropriate.
Environment
- codex-cli 0.142.5, Linux (Ubuntu 22.04), standalone install
~/.codexon local disk; hundreds of live + 507 archived sessions- Trigger correlated with a root-filesystem ENOSPC event (0 bytes free) during an archival burst
Activity
github-actions commentedon Jul 4, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
ideajoker commentedon Jul 8, 2026
版本 26.623.141536 • 发布于 2026年7月7日
no rollout found for thread id
Odezra commentedon Jul 9, 2026
Additional macOS evidence showing a one-for-one state transition after a legacy filesystem-level archive move:
0.143.00.144.0-alpha.4OKBefore and after
codex doctor --jsoncounts changed exactly with the moved set:This supports the report's hypothesis that filesystem-level archival can leave both sides of the inventory inconsistent: the preserved archived rollout is unindexed, while the DB row remains at the old active path.
No manual SQLite repair or speculative filesystem rollback has been attempted. The archive workflow is paused pending a supported recovery path.
Could Codex expose a previewable reconcile/reindex command that:
codex doctor --json.A full redacted Doctor report can be provided privately if useful.
xiaodeng2025 commentedon Aug 18, 2026
Windows Desktop confirmation, with a newer app package.
OpenAI.Codex_26.814.5167.0_x64Observed behavior:
thread not foundand it cannot be continued.Exact archive error:
One affected thread ID:
01a013b2-9800-75d0-ae83-35e9dbc50a8eThe task record is still readable through the local task listing, and its workspace directory still exists. I have not performed a filesystem or SQLite audit, so I cannot confirm that this is the same missing-rollout mechanism as the original report; it is a symptom-level match. I have deliberately not reset the app, cleared
.codex, or deleted the workspace, to avoid making recoverable local history less recoverable.A safe, previewable reconcile/reindex tool or an option to remove an orphaned sidebar entry would be especially useful here.
xulfereht commentedon Aug 31, 2026
Confirming this is still present on CLI
0.151.0(current as of 2026-08-31).codex doctorsurfaces it as a first-class warning now:on a host with 630 active rollout files (6.84 GiB) — so
doctoris aware of the inconsistency but doesn't offer a--fix/reconcile action, and I couldn't find one elsewhere in the CLI (migrate-rolloutsinspects/migrates legacy rollouts into paginated thread history, it doesn't reconcile stale index rows). Acodex doctor --fixor a dedicatedcodex reindex-style command that drops/repairs stalesession_index/thread-store rows for files confirmed missing (with the archived-copy fallback this issue already asks for) would close the loop doctor's own diagnostic opens.kip-claw commentedon Oct 14, 2026
Follow-up with a much more extreme ratio than previously reported, plus a downstream consequence worth flagging: this appears to directly cause a significant startup-performance regression when codex runs as an embedded harness inside another process (OpenClaw).
Scale
94.6% stale is a much higher ratio than the 40% in the original report — on a host that's been in continuous daily use (not crashed-and-abandoned), with rollouts spanning a continuous ~2-month window (Aug 19 - Oct 14) with no gap.
codex doctorcorrectly surfaces this as⚠ threads state DB rows point at missing or unusable rollout files, matching the warning text quoted in earlier comments here.Confirms the proposed crash-inconsistency root cause
This host has experienced, independently and repeatedly over the measurement period: multiple
ENOSPC: no space left on deviceevents (from an unrelated upstream openclaw/openclaw bug that fills a shared tmp directory — openclaw/openclaw#152689), and at least one hardSIGKILLof the parent process group during an unresponsive shutdown (openclaw/openclaw's own supervisor escalating past a gracefulSIGTERMbecause the process was too busy to respond in time). Both match the disk-full/SIGKILL triggers this issue's root-cause theory identifies. Given how consistently this host hits exactly those trigger conditions, a 94.6% stale ratio over ~2 months of real usage seems like strong supporting evidence for "nothing later reconciles it" — the index just keeps accumulating dead entries indefinitely with no decay or self-heal.No officially-supported recovery path found
I looked for a safe way to clean this up before filing:
codex doctor— diagnoses only, no--fix.codex migrate-rollouts --apply— ran cleanly (186 migrated, 0 failed), but addresses a different concern (legacy flat-file format → paginated format) and left the stale-row count andstate_5.sqlitefile size completely unchanged (993MB before and after).archive/delete/vacuum/gc command exists;archive/deleteonly operate one session at a time, and these stale rows don't correspond to any real session to target anyway.state_5.sqlite— we've been burned before by in-place modification of a sibling codex database (logs_2.sqlite) desyncing its sqlx migration state and breaking app-server initialization, so improvising a fix onstate_5.sqlitewithout guidance felt too risky.Downstream consequence: this is a measurable factor in slow Gateway cold starts
For context (full details in openclaw/openclaw#162211, a startup-performance issue in the OpenClaw project that embeds codex as a harness): on this same host, the
codexplugin is one of three official plugins responsible for ~90 of ~123 seconds of Gateway cold-start time, with the embedding process's own event loop fully blocked for the duration.state_5.sqliteat 993MB is one of the largest filescodextouches at startup, and its size is apparently driven almost entirely by this stale-row accumulation rather than real session volume (1,033 real files vs. 19,046 indexed rows). I can't prove the stale rows are the cause of codex's slow load specifically — I don't have a way to compare against a clean index on this host — but it's a very plausible contributor, and a decay/cleanup mechanism for this index would likely help both issues at once.Happy to share the full
codex doctor --jsonoutput (redacted) if useful.kip-claw commentedon Oct 15, 2026
Follow-up to my comment above: I tested the hypothesis directly, and it's wrong — posting the negative result so it doesn't mislead anyone chasing the same lead.
With the Gateway stopped, I moved
state_5.sqlite(993MB) andthread_history_1.sqlite(394MB) aside entirely and letcodexrecreate them fresh on next start (confirmed viacodex doctorafterward: clean index, 0 stale rows). Measured thecodexplugin's load time inside the embedding Gateway across three restarts before and after:Total Gateway cold-start time was also unchanged (118s vs. 116-138s baseline). So the stale-row bloat in
state_5.sqliteis real and worth fixing on its own merits, but it is not the cause ofcodex's slow plugin-load time when embedded in OpenClaw — that bottleneck is somewhere else. Restored the original files afterward (no reason to lose session history for a fix that didn't work).Didn't mean to send anyone down a dead end — wanted to correct the record once I had the data.