Codex stale-thinking spinner persists after stream state desync (thread stale detection + targeted recovery results)

Reproducibility summary

Date: 2026-02-26 (UTC)
Scope: local sessions under ~/.codex/sessions, ~/.codex/archived_sessions, app logs under ~/Library/Logs/com.openai.codex.
Detector is read-only for discovery; recovery attempts are targeted per-thread only.
This issue is tracked with generated artifacts under:
- /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221328Z/
No raw conversation text or secrets are included in the artifacts.

Key stale indicators

019c9a7f-46f2-7e03-aceb-89afe89a581b => stale score=12.6, unmatched_starts=2, heartbeat_ms=4650543, in_progress_stream=true, marked_streaming=true, turn_not_found_count=1272.
019c489a-0eb0-7142-8c30-e92ced2cf74a => at-risk score=5.8, unmatched_starts=0, heartbeat_ms=8332044, in_progress_stream=true, marked_streaming=true, turn_not_found_count=0.

Recovery results from tooling

Targeted conservative recovery (--recover) was executed per-thread after detection on the two above threads.
For both threads, the app-server returned RPC error: thread not found during turn/interrupt.
Recovery status was tracked as thread_not_found; operator guidance is to continue in a fresh thread using a compact handoff of last confirmed artifacts.

Observable loop behavior

Thread UI remains in thinking/streaming indications after maybe_resume_success appears with latestTurnStatus=inProgress and markedStreaming=true.
Turn not found for event and thread-stream-state-changed / handler warning patterns continue in logs around those thread IDs.
This suggests a stream-state reconciliation or state-database migration/cleanup race rather than pure compute inactivity.

Ask

Please investigate stream-state lifecycle for thread-stream-state-changed, maybe_resume_success, and Turn not found for event reconciliation.
Confirm whether stale stream markers should be auto-cleared when referenced turns are no longer resolvable.

Follow-up from production-hardening run:

Run: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221428Z/
Target threads:
- 019c9a7f-46f2-7e03-aceb-89afe89a581b
- 019c489a-0eb0-7142-8c30-e92ced2cf74a
Tool classification:
- stale and at-risk respectively (heartbeat > 1h, stream markers present)
Recovery attempt (automated, thread-safe): thread_not_found for both IDs on turn/interrupt.
Script now treats this as terminal thread_not_found state and emits operator fallback guidance rather than generic failure.
Recommendation remains: continue in a fresh thread using compact handoff of last confirmed completed steps.

Artifacts in this run include:

stale_threads.jsonl
actions.json
incident_summary.md
redacted_session_map.csv
openai_feedback_draft.md
recovery_results.json

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

Codex stale-thinking thread spinner may persist after stream-state mismatch #12954

Powered by Codex Action

Follow-up update from latest run:
- Added phrase-signal escalation path in detector (you are stuck somewhere. stop and replan / Reprise: j’ai relancé un test MLX), and explicit per-thread recovery forcing for --recover-thread targets.
- Latest run: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T221609Z/
- 019c9ba7-e168-78a1-97f2-a188bc74d44a also now surfaces as targeted recovery candidate (with phrase-signal evidence), but app-server returns thread_not_found when interrupted.
- Recovery outputs remain per-thread only, non-global; no unrelated threads are stopped.

Follow-up production-hardening result (run ):

Script: [stale-think] stale_threads=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/stale_threads.jsonl
[stale-think] actions=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/actions.json
[stale-think] summary=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/incident_summary.md
[stale-think] map=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/redacted_session_map.csv
[stale-think] openai_feedback=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/openai_feedback_draft.md
[stale-think] github_payload=/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222114Z/github_issue_payload.json
[stale-think] stale_count=2 total_threads=57
Targets: , ,
Detection outcome: / / phrase-linked candidate.
Recovery outcome: all three return on with required operator path: or (fresh thread + compact handoff).
Version matrix is now explicitly tracked and this run shows mismatch: vs ().
Evidence artifacts in including recovery results, stale rows, and final report.
Automation remains active: (, no global stop actions).

Follow-up production-hardening result (run 20260226T222100Z):

Script: /Users/jeremie/.codex/scripts/codex_stale_thinking_monitor.py
Targets: 019c9a7f-46f2-7e03-aceb-89afe89a581b, 019c489a-0eb0-7142-8c30-e92ced2cf74a, 019c9ba7-e168-78a1-97f2-a188bc74d44a
Detection outcome: stale, at-risk, phrase-linked candidate.
Recovery outcome: all three return thread_not_found on turn/interrupt with required operator path: resume? if still visible, otherwise start fresh thread with compact handoff.
Version alignment: codex_cli 0.101.0 vs codex_app 0.105.0 (mismatch).
Evidence artifacts: /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222100Z/ (recovery_results, stale_threads, openai_feedback_draft, incident_summary).
Automation remains active: /Users/jeremie/.codex/automations/codex-stale-think-watch/automation.toml (hourly, no global stop actions).

Updated follow-up packet (run 20260226T222239Z):

Recovery remains blocked for primary stale candidate 019c9a7f-46f2-7e03-aceb-89afe89a581b and at-risk 019c489a-0eb0-7142-8c30-e92ced2cf74a in this execution window.
Final RC package now in /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T222239Z/final_rc_report.md.
Version mismatch continues: codex_cli 0.101.0 vs codex_app 0.105.0 (version_alignment: mismatch).
This run adds explicit operator handoff language in tooling: Recovered thread: resume? or new (fresh thread handoff). Automation prompt updated accordingly in automations/codex-stale-think-watch/automation.toml.

Follow-up production-hardening run after CLI/App alignment:\n\n- Aligned CLI to app protocol version via
changed 2 packages in 435ms (now both report ).\n- Re-ran targeted recovery on: and .\n- Result remains terminal for both: app-server returns on ; status in .\n- Operator handoff remains explicit: if thread still visible in UI, otherwise to continue from compact handoff.\n- Artifacts for this run:\n - stale_threads.jsonl\n - actions.json\n - openai_feedback_draft.md\n - recovery_results.json\n - incident_summary.md\n - github_issue_payload.json\n\nCurrent blocker for closure is persisted stream-state artifact for these IDs, likely orphaned run IDs/state-db mismatch independent of protocol version skew.

Follow-up production-hardening run after CLI/App alignment:

Aligned CLI to app protocol version via npm install -g @openai/codex@0.105.0 (now both report codex-cli 0.105.0).
Re-ran targeted recovery on thread IDs:
- 019c9a7f-46f2-7e03-aceb-89afe89a581b
- 019c489a-0eb0-7142-8c30-e92ced2cf74a
Result remains terminal for both: app-server returns thread not found on turn/interrupt; status thread_not_found in /Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T230417Z/recovery_results.json.
Operator handoff remains explicit: Recovered thread: resume? if thread still visible in UI, otherwise new for compact handoff in a fresh thread.
Artifacts for this run:
- stale_threads.jsonl
- actions.json
- openai_feedback_draft.md
- recovery_results.json
- incident_summary.md
- github_issue_payload.json

Current blocker for closure is persisted stream-state artifacts for these IDs, likely orphaned run IDs/state-db mismatch independent of protocol version skew.

RC-ready summary posted in local packet:
/Users/jeremie/.codex/reports/codex-stale-thinking/2026-02-26/20260226T230417Z/final_rc_report.md

This confirms both flagged threads are in final state:

thread_not_found on targeted recovery attempts
safe recovery mode required operator handoff only
explicit prompt available: Recovered thread: resume? or new

CLI/App are now aligned at 0.105.0 (npm list -g @openai/codex --depth=0), so remaining blocker is stale/orphaned thread state in old session artifacts.

Follow-up (2026-02-27, production rollout + evidence):

We implemented and validated an automated stale-think containment path in local production.

What is now automated:

Detect stale threads from logs/transcripts using:

unmatched task_started/task_complete
inProgress+markedStreaming markers
turn reconciliation errors ("thread not found", "Turn not found for event")
heartbeat gap

Conservative per-thread recovery first (no global restart).
If recovery fails (thread_not_found / still stale), auto-relaunch to a fresh thread in Plan mode with continuity context and explicit ops guidance bootstrap.
Crash protection in the same loop:

detects Chrome runaway memory by total RSS and peak process RSS
optional peak-process termination to prevent system hangs

Operational evidence (sanitized):

Source stale threads:
- 019c489a-0eb0-7142-8c30-e92ced2cf74a
- 019c9a7f-46f2-7e03-aceb-89afe89a581b
Recovery attempts on stale source threads return thread_not_found even while stale indicators remain in historical telemetry.
Auto-relaunch created fresh continuation threads successfully:
- library continuation: 019c9fee-0556-7173-9509-d6f3b64eaef8 (turn completed)
- openclaw continuation: 019c9fee-0555-77f0-b351-9ba90c013fd0 (turn entered inProgress then continued)

Updated hypothesis:

stale spinner appears to be a UI/state reconciliation artifact after stream desync.
old thread IDs can become unreachable to app-server (thread not found) while UI/log-derived stale state still persists for that thread lineage.
fallback relaunch is effective for continuity, but root cause remains in stream-state reconciliation / orphaned turn handling.

Requested backend/UI focus:

hard reconciliation when source thread is not resolvable but UI still marks streaming.
idempotent closure for orphaned inProgress turn markers.
stronger handling for thread-stream-state-changed + missing turn references (Turn not found for event).

Sanitized artifact roots used for this validation:

~/.codex/reports/codex-stale-thinking/2026-02-27/
~/.codex/runtime/relaunched_threads.jsonl

If useful, I can post a compact event timeline matrix keyed by (thread_id, timestamp, event_type) without prompt contents.

I presume this is a bot. Closing.

Additional follow-up (same day):

New high-signal anomaly observed while hardening fallback logic:

thread/read succeeds for affected thread IDs, but turn/interrupt on the same (threadId, turnId) returns thread not found (-32600).
This was reproducible on previously stale source threads and on some freshly relaunched threads.

Example behavior class (sanitized):

thread/read(threadId=X, includeTurns=true) => returns turns incl. inProgress.
turn/interrupt(threadId=X, turnId=Y) => thread not found: X.
UI can remain in spinner state despite inability to interrupt.

Operational mitigation added:

If relaunch turn remains stuck inProgress, auto-archive that relaunch thread.
Create/attempt checkpoint continuation; if checkpoint also hangs, archive it too to avoid phantom spinner accumulation.
Keep only non-spinning continuation threads visible.

Result from latest validation window:

originally flagged source IDs no longer present as stale in active queue.
one unrelated stale lineage remains under monitor.

This seems like a request-routing/state-index inconsistency between read-path and interrupt-path for thread identity resolution.

Navigation Menu

Repository navigation

Codex stale-thinking spinner persists after stream state desync (thread stale detection + targeted recovery results) #12955

Reproducibility summary

Key stale indicators

Recovery results from tooling

Observable loop behavior

Ask

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Codex stale-thinking spinner persists after stream state desync (thread stale detection + targeted recovery results) #12955

Description

Reproducibility summary

Key stale indicators

Recovery results from tooling

Observable loop behavior

Ask

Activity

g0udurix commented on Feb 26, 2026

github-actions commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 26, 2026

g0udurix commented on Feb 27, 2026

etraut-openai commented on Feb 27, 2026

g0udurix commented on Feb 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions