Skip to content

codex-rescue agent's contradictory background contract runs the companion in foreground — job orphaned at subagent turn end, permanently stuck at 'running' #486

Open

Description

@tjcs

Bug report: codex-rescue agent's contradictory background contract runs the companion in foreground — job orphaned at subagent turn end, permanently stuck at running

Component: openai-codex Claude Code plugin v1.0.6 (commit db52e28), agents/codex-rescue.md + scripts/codex-companion.mjs + scripts/lib/{tracked-jobs,job-control}.mjs
Platform: macOS (darwin 24), Node v20.19.6, codex-cli 0.144.1, Claude Code harness
Severity: High — any substantial task routed through the codex:codex-rescue agent silently loses its result; the job record is stuck at status: "running" forever and /codex:result can never retrieve it.

Related: #478 (no PID-liveness reaping — the persistence half of this failure), #458 (state.json race). This issue is about the trigger: the rescue agent's execution contract makes the orphan inevitable.

What happened

A Claude Code session dispatched a large read-only PR review to the codex:codex-rescue subagent. Three attempts in one session, all lost the same way:

Attempt Lifetime How it died
1 ~117s foreground task call killed at the host's Bash tool timeout (~120s)
2 ~56s+ same, killed before the next attempt started
3 (pid 3602) ~174s agent wrapped the foreground task call in a host-side background shell, then ended its turn; the harness reaped the shell and the companion with it

Attempt 3 completed its entire read phase (git diff, every file) and died mid-turn before any final message. Per the agent's instructions it reported nothing about attempts 1–2, and for attempt 3 told its caller "I'll be notified when it completes" — a notification mechanism that does not exist across a completed subagent's boundary.

End state: job JSON frozen at status: "running" with a dead pid, /codex:status shows a phantom running job indefinitely, /codex:result throws Job … is still running forever.

Forensic evidence that the foreground path was used

The orphaned job record has no request field and never passed through queued — both are written only by enqueueBackgroundTask() (scripts/codex-companion.mjs ~L684–697). And the host-side task log contains the [codex]-prefixed stderr progress stream that only runForegroundCommand() emits (stderr: !options.json, ~L658–663). So the properly daemonized worker path — spawnDetachedTaskWorker() with detached: true, stdio: "ignore", unref() (~L671–682), which would have survived every teardown — was never engaged.

Root cause: every instruction-compliant path fails

agents/codex-rescue.md gives the model a contract with no valid move for a long task:

  • L23–24: prefer foreground for small tasks, "prefer background execution" for complicated/long ones.
  • But --background is defined as Claude-side control (commands/rescue.md ~L16) and the runtime skill orders it stripped before calling task (skills/codex-cli-runtime/SKILL.md ~L26) — so the agent never passes --background to the companion.
  • L27–28: the agent must not "monitor progress, poll status, fetch results" and must not call status, result, or cancel — so the plugin's own launch → poll → fetch machinery (status --wait, /codex:result) is forbidden to it.
  • L42: "If the Bash call fails or Codex cannot be invoked, return nothing" — which silently swallows the timeout kills.

Foreground blocking dies at the host Bash timeout for any real task; companion-level background is stripped; polling is banned. The model improvises the one remaining option — host-side backgrounding of a foreground companion — which is the worst case: lifetime-coupled to the subagent's turn, with nobody left to poll.

Two mechanical defects then make the loss permanent:

  1. No signal handling in the companion. runTrackedJob() (scripts/lib/tracked-jobs.mjs ~L142–203) reaches a terminal status only via normal completion or a caught exception; SIGTERM bypasses the only writer of status: "failed".
  2. No dead-worker reconciliation (as in codex-companion background jobs: no turn timeout, no PID-liveness reaping, shell-mangled taskkill (Windows) — jobs stick as 'running' forever #478): resolveResultJob() (scripts/lib/job-control.mjs ~L256–271) accepts only completed/failed/cancelled and reports active records as "still running" — permanently, since nothing ever flips them.

A related race worth noting while in this code: enqueueBackgroundTask() spawns the detached worker before persisting the job/request file, so the worker can win the race, find no request, and exit — leaving a permanently queued record.

Suggested fix

Short term (documentation-level, no runtime changes):

  • Rewrite agents/codex-rescue.md so the agent launches non-trivial tasks with task --background, then loops bounded status <jobId> --wait --timeout-ms 120000 calls until the job is terminal, then returns result <jobId> output verbatim. Permit status/result for the job the agent itself launched. Replace "return nothing" with returning the job id plus failure detail. Explicitly forbid host-side backgrounding of a foreground task call.

Longer term (runtime):

Workaround for affected users

The Codex thread usually survives the companion's death. Read threadId from the stuck job JSON under ~/.claude/plugins/data/codex-openai-codex/state/<workspace-hash>/jobs/ and resume it directly:

codex exec --skip-git-repo-check --sandbox read-only -o /tmp/out.md --color never resume <threadId> "Finish the task and emit your final answer."

For write-capable jobs, confirm the original turn is actually dead first (a broker-hosted turn can outlive its companion) to avoid racing it.


Diagnosed with Claude Code against the v1.0.6 source and on-disk job records; analysis independently cross-checked with the Codex CLI (codex exec, read-only) against the same source. Line numbers are from the v1.0.6 release and marked approximate (~) where the reviewer should re-anchor.

Activity

  1. axisrow commented on Jul 14, 2026

    @axisrow

    The mechanical defects (no signal handling in runTrackedJob, no dead-worker reconciliation) are real, but the deeper blocker is that this failure is invisible even after the fact — exactly what #481 documents. The orphaned job record stores no resolved model/effort, and the only corpse evidence lives in the rollout under a session-id that doesn't match the job's threadId (I verified this join fails outright against codex-cli 0.144.1). So even if the rescue agent's foreground contract is fixed, there's currently no way for a caller to prove a dispatched rescue task actually ran vs. died silently — /codex:result just says "still running" forever. Fixing #481 (persist resolved model/effort + a terminal worker_died status from the liveness reaping in #478) is what makes this class of loss diagnosable. The agent-contract rewrite is necessary but not sufficient without it.

  2. sjarmak commented on Jul 16, 2026

    @sjarmak

    Confirming this on Linux (ds-5090) with plugin v1.0.2 driving codex-cli through the app-server broker, so the failure spans both the OS and the plugin version in the original report.

    Same signature, captured from the job state files after two stalled codex:codex-rescue reviews in one session:

    • state.json job frozen at status: running / phase: running, updatedAt about 9s after startedAt, never terminal.
    • The app-server-broker.mjs broker PID and the task worker PID both dead.
    • broker.log empty (0 bytes), i.e. SIGKILL with no graceful shutdown line written.
    • The job log stops streaming at the exact second the forwarder subagent completes its turn.

    The companion daemonizes correctly (detached: true + unref() in broker-lifecycle.mjs and spawnDetachedTaskWorker), but that only escapes the process group, not the cgroup/scope the harness tears down when the subagent ends, so the detached broker and worker are reaped anyway.

    Workaround that survives, for anyone blocked: drive the companion's own durable path from the main session rather than the rescue subagent, so the enqueue shell is never reaped mid-turn.

    node codex-companion.mjs task --background --json "<prompt>"    # detached worker, returns jobId
    node codex-companion.mjs status <jobId> --wait --timeout-ms N   # blocks until terminal
    node codex-companion.mjs result <jobId> --json                  # rawOutput
    

    status --wait and result both read the on-disk job state, so the poller is disposable while the detached worker is the durable unit; a killed poller can re-attach with status/result. This completed a roughly 3.5-minute read-only engine-core review that the foreground rescue path had stalled on twice.

  3. stefan8 commented on Jul 30, 2026

    @stefan8

    Confirming on 1.0.6. The contradiction is still there: agents/codex-rescue.md:23 says to prefer background for work "likely to keep Codex running for a long time", while commands/rescue.md:18 says "If neither flag is present, default to foreground". In practice the command text wins, so an unflagged code review — open-ended by nature — gets dispatched in foreground and dies at Claude Code's 600 s Bash ceiling (timing data in #542: my killed run stopped at 599.7 s, while every --background run in the same install completed at 1048–2663 s).

    The orphaning then compounds it: the record wedges at status: "running", and because findLatestResumableTaskJob() skips running jobs, the threadId sitting in that same record cannot be used to resume.

    Duration is not knowable up front, so guidance alone can't resolve this — the safe default for task dispatch is background, with foreground reserved for explicitly bounded calls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions