Skip to content

Detached task-worker exits without writing failed status on broker socket disconnect #21813

Open

Description

@vstandos

Issue

When a codex-companion.mjs task --background task-worker process exits due to broker JSON-RPC socket close/reset, runTrackedJob's catch block is never reached. The job stays permanently in status: "running" in state.json with a now-dead PID.

Reproduce

  1. Start a long-running codex task in background mode
  2. Kill the broker mid-task OR cause broker socket reset (e.g., transient network blip on app-server connection)
  3. task-worker process dies silently
  4. codex status reports "running" indefinitely

Evidence (from local instance)

  • task-worker spawn: codex-companion.mjs lines 641-679 (spawnDetachedTaskWorker with detached: true, stdio: "ignore")
  • State write on completion: lib/tracked-jobs.mjs lines 153-203 — only catch path writes failed status
  • Two ghost-running tasks observed today with last activity timestamps minutes before broker reset, then no log entries, no error, dead PIDs

Proposed fix

Add process.on('exit'/'uncaughtException'/'unhandledRejection') handler in handleTaskWorker that synchronously writes status: "failed" to the job file before exit. Without this, operators cannot distinguish actively-running tasks from dead ones.

Workaround

Operators must ps -p <PID> to check if the worker is alive. If dead, manually mark task as failed in state.json.

Environment: codex-cli 0.129.0, Linux x86_64 (WSL2 Ubuntu 24.04), npm 11.x.

Activity

  1. added
    bugSomething isn't working
    CLIIssues related to the Codex CLI
    app-serverIssues involving app server protocol or interfaces
    connectivityIssues involving networking or endpoint connectivity problems (disconnections)
    on May 8, 2026
  2. geoff-ux commented on Jun 7, 2026

    @geoff-ux

    Additional local evidence from the Claude Code Codex plugin path.

    Environment:

    • macOS / Darwin arm64
    • Claude Code Codex plugin: openai-codex/codex 1.0.2
    • codex --version: codex-cli 0.137.0
    • Codex.app embedded CLI: codex-cli 0.136.0-alpha.2

    Observed symptom:

    • /codex:status showed a background task as running, phase starting (8m 39s, no activity after turn-start).
    • The job state still had status: "running", phase: "starting", a stored PID, thread id, and turn id.
    • ps -p <stored-pid> showed the worker process was already gone.
    • The per-job progress log only contained:
    Starting Codex Task.
    Starting Codex task thread.
    Thread ready (...)
    Turn started (...)
    
    • The corresponding ~/.codex/sessions/.../rollout-*.jsonl had only session_meta and task_started, with no assistant message, tool call, error, or terminal event.
    • A scan of the plugin state found six running/queued jobs with stored PIDs that no longer existed.

    This makes the user-visible failure look like a silent model/runtime stall after turn-start, but at least some of these are stale companion records for already-dead detached workers.

    One extra UX/tooling angle: scripts/lib/job-control.mjs enriches and renders job state from state.json/progress lines, but does not appear to liveness-check the stored PID before presenting the task as running. Even if the worker death path is fixed, a defensive status/result path that marks or reports stale/dead-pid would make this much easier to diagnose and would avoid operators waiting on jobs that cannot make progress.

    Related local code observations from plugin 1.0.2:

    • scripts/lib/tracked-jobs.mjs writes status: "running"/phase: "starting" with pid: process.pid before running the task.
    • Terminal status is written on normal runner completion or the runner catch path.
    • createJobProgressUpdater() only updates persisted state when phase/thread/turn changes, so a task can show old updatedAt while still appearing running.
    • scripts/lib/job-control.mjs infers starting from the last progress lines and renders elapsed time, but does not distinguish live worker vs dead PID.

    This was not caused by a local startup hook in this case. A repo hook did add about 10 seconds before useful work, but the dead record remained running for 8+ minutes with no live process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIapp-serverIssues involving app server protocol or interfacesbugSomething isn't workingconnectivityIssues involving networking or endpoint connectivity problems (disconnections)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions