Repository navigation
codex-rescue agent's contradictory background contract runs the companion in foreground — job orphaned at subagent turn end, permanently stuck at 'running' #486
Description
Bug report: codex-rescue agent's contradictory background contract runs the companion in foreground — job orphaned at subagent turn end, permanently stuck at running
Component: openai-codex Claude Code plugin v1.0.6 (commit db52e28), agents/codex-rescue.md + scripts/codex-companion.mjs + scripts/lib/{tracked-jobs,job-control}.mjs
Platform: macOS (darwin 24), Node v20.19.6, codex-cli 0.144.1, Claude Code harness
Severity: High — any substantial task routed through the codex:codex-rescue agent silently loses its result; the job record is stuck at status: "running" forever and /codex:result can never retrieve it.
Related: #478 (no PID-liveness reaping — the persistence half of this failure), #458 (state.json race). This issue is about the trigger: the rescue agent's execution contract makes the orphan inevitable.
What happened
A Claude Code session dispatched a large read-only PR review to the codex:codex-rescue subagent. Three attempts in one session, all lost the same way:
| Attempt | Lifetime | How it died |
|---|---|---|
| 1 | ~117s | foreground task call killed at the host's Bash tool timeout (~120s) |
| 2 | ~56s+ | same, killed before the next attempt started |
| 3 (pid 3602) | ~174s | agent wrapped the foreground task call in a host-side background shell, then ended its turn; the harness reaped the shell and the companion with it |
Attempt 3 completed its entire read phase (git diff, every file) and died mid-turn before any final message. Per the agent's instructions it reported nothing about attempts 1–2, and for attempt 3 told its caller "I'll be notified when it completes" — a notification mechanism that does not exist across a completed subagent's boundary.
End state: job JSON frozen at status: "running" with a dead pid, /codex:status shows a phantom running job indefinitely, /codex:result throws Job … is still running forever.
Forensic evidence that the foreground path was used
The orphaned job record has no request field and never passed through queued — both are written only by enqueueBackgroundTask() (scripts/codex-companion.mjs ~L684–697). And the host-side task log contains the [codex]-prefixed stderr progress stream that only runForegroundCommand() emits (stderr: !options.json, ~L658–663). So the properly daemonized worker path — spawnDetachedTaskWorker() with detached: true, stdio: "ignore", unref() (~L671–682), which would have survived every teardown — was never engaged.
Root cause: every instruction-compliant path fails
agents/codex-rescue.md gives the model a contract with no valid move for a long task:
- L23–24: prefer foreground for small tasks, "prefer background execution" for complicated/long ones.
- But
--backgroundis defined as Claude-side control (commands/rescue.md~L16) and the runtime skill orders it stripped before callingtask(skills/codex-cli-runtime/SKILL.md~L26) — so the agent never passes--backgroundto the companion. - L27–28: the agent must not "monitor progress, poll status, fetch results" and must not call
status,result, orcancel— so the plugin's own launch → poll → fetch machinery (status --wait,/codex:result) is forbidden to it. - L42: "If the Bash call fails or Codex cannot be invoked, return nothing" — which silently swallows the timeout kills.
Foreground blocking dies at the host Bash timeout for any real task; companion-level background is stripped; polling is banned. The model improvises the one remaining option — host-side backgrounding of a foreground companion — which is the worst case: lifetime-coupled to the subagent's turn, with nobody left to poll.
Two mechanical defects then make the loss permanent:
- No signal handling in the companion.
runTrackedJob()(scripts/lib/tracked-jobs.mjs~L142–203) reaches a terminal status only via normal completion or a caught exception; SIGTERM bypasses the only writer ofstatus: "failed". - No dead-worker reconciliation (as in codex-companion background jobs: no turn timeout, no PID-liveness reaping, shell-mangled taskkill (Windows) — jobs stick as 'running' forever #478):
resolveResultJob()(scripts/lib/job-control.mjs~L256–271) accepts onlycompleted/failed/cancelledand reports active records as "still running" — permanently, since nothing ever flips them.
A related race worth noting while in this code: enqueueBackgroundTask() spawns the detached worker before persisting the job/request file, so the worker can win the race, find no request, and exit — leaving a permanently queued record.
Suggested fix
Short term (documentation-level, no runtime changes):
- Rewrite
agents/codex-rescue.mdso the agent launches non-trivial tasks withtask --background, then loops boundedstatus <jobId> --wait --timeout-ms 120000calls until the job is terminal, then returnsresult <jobId>output verbatim. Permitstatus/resultfor the job the agent itself launched. Replace "return nothing" with returning the job id plus failure detail. Explicitly forbid host-side backgrounding of a foregroundtaskcall.
Longer term (runtime):
- Make every task execute in a durable (detached) worker; "foreground" should mean wait on that worker, so the caller never owns execution. Persist the job/request atomically before spawn and require a startup ack.
- Install SIGTERM/SIGINT handlers that terminalize the job record (and best-effort interrupt the Codex turn) before exit.
- Reconcile dead workers on the
status/result/resumeread paths (per codex-companion background jobs: no turn timeout, no PID-liveness reaping, shell-mangled taskkill (Windows) — jobs stick as 'running' forever #478), guarding against pid reuse.
Workaround for affected users
The Codex thread usually survives the companion's death. Read threadId from the stuck job JSON under ~/.claude/plugins/data/codex-openai-codex/state/<workspace-hash>/jobs/ and resume it directly:
codex exec --skip-git-repo-check --sandbox read-only -o /tmp/out.md --color never resume <threadId> "Finish the task and emit your final answer."
For write-capable jobs, confirm the original turn is actually dead first (a broker-hosted turn can outlive its companion) to avoid racing it.
Diagnosed with Claude Code against the v1.0.6 source and on-disk job records; analysis independently cross-checked with the Codex CLI (codex exec, read-only) against the same source. Line numbers are from the v1.0.6 release and marked approximate (~) where the reviewer should re-anchor.
Activity
axisrow commentedon Jul 14, 2026
The mechanical defects (no signal handling in
runTrackedJob, no dead-worker reconciliation) are real, but the deeper blocker is that this failure is invisible even after the fact — exactly what #481 documents. The orphaned job record stores no resolved model/effort, and the only corpse evidence lives in the rollout under a session-id that doesn't match the job'sthreadId(I verified this join fails outright against codex-cli 0.144.1). So even if the rescue agent's foreground contract is fixed, there's currently no way for a caller to prove a dispatched rescue task actually ran vs. died silently —/codex:resultjust says "still running" forever. Fixing #481 (persist resolved model/effort + a terminalworker_diedstatus from the liveness reaping in #478) is what makes this class of loss diagnosable. The agent-contract rewrite is necessary but not sufficient without it.sjarmak commentedon Jul 16, 2026
Confirming this on Linux (ds-5090) with plugin v1.0.2 driving codex-cli through the app-server broker, so the failure spans both the OS and the plugin version in the original report.
Same signature, captured from the job state files after two stalled
codex:codex-rescuereviews in one session:state.jsonjob frozen atstatus: running/phase: running,updatedAtabout 9s afterstartedAt, never terminal.app-server-broker.mjsbroker PID and thetaskworker PID both dead.broker.logempty (0 bytes), i.e. SIGKILL with no graceful shutdown line written.The companion daemonizes correctly (
detached: true+unref()inbroker-lifecycle.mjsandspawnDetachedTaskWorker), but that only escapes the process group, not the cgroup/scope the harness tears down when the subagent ends, so the detached broker and worker are reaped anyway.Workaround that survives, for anyone blocked: drive the companion's own durable path from the main session rather than the rescue subagent, so the enqueue shell is never reaped mid-turn.
status --waitandresultboth read the on-disk job state, so the poller is disposable while the detached worker is the durable unit; a killed poller can re-attach withstatus/result. This completed a roughly 3.5-minute read-only engine-core review that the foreground rescue path had stalled on twice.stefan8 commentedon Jul 30, 2026
Confirming on 1.0.6. The contradiction is still there:
agents/codex-rescue.md:23says to prefer background for work "likely to keep Codex running for a long time", whilecommands/rescue.md:18says "If neither flag is present, default to foreground". In practice the command text wins, so an unflagged code review — open-ended by nature — gets dispatched in foreground and dies at Claude Code's 600 s Bash ceiling (timing data in #542: my killed run stopped at 599.7 s, while every--backgroundrun in the same install completed at 1048–2663 s).The orphaning then compounds it: the record wedges at
status: "running", and becausefindLatestResumableTaskJob()skipsrunningjobs, thethreadIdsitting in that same record cannot be used to resume.Duration is not knowable up front, so guidance alone can't resolve this — the safe default for
taskdispatch is background, with foreground reserved for explicitly bounded calls.