Skip to content

codex-companion background jobs: no turn timeout, no PID-liveness reaping, shell-mangled taskkill (Windows) — jobs stick as 'running' forever #478

Open

Description

@bilclinti

Bug report: codex-companion job runner has no turn timeout, no PID-liveness check, and a shell-mangling taskkill call

Component: openai-codex Claude Code plugin, version 1.0.6, scripts/codex-companion.mjs + scripts/lib/{tracked-jobs,state,process}.mjs
Platform: Windows (win32), jobs launched via the plugin's task --background / rescue code path
Severity: High — a single wedged turn permanently occupies a scheduler slot and is invisible to status as "stuck"; only a process-tree kill + manual state.json surgery recovers it.

Bug 1 — no wall-clock timeout on a tracked job's turn

runTrackedJob() (scripts/lib/tracked-jobs.mjs:142) does:

const execution = await runner();

with no timer, AbortController, or any other mechanism racing the runner. If the underlying Codex turn wedges — e.g. a shell child command it spawned mid-turn hangs (stdin wait, dead network call, orphaned handle) — the awaited promise never resolves and the job record never leaves status: "running".

Evidence: a real job was observed frozen at phase: "verifying", with its last recorded progress event being a custom_tool_call whose output never arrived (the child that should have produced it never returned). By the time it was noticed, the thread had accumulated ~57M input tokens — a runaway loop re-feeding stale context with no timeout to cut it off.

Bug 2 — no PID-liveness check when jobs are read back from state.json

listJobs() (scripts/lib/state.mjs:149) returns the jobs array from state.json verbatim:

export function listJobs(cwd) {
  return loadState(cwd).jobs;
}

There is no check anywhere in the read path that a job reported running/queued still has a live process behind its pid. If the worker process dies (terminal closed, Ctrl+C, crash, reboot) without going through the normal completion path in runTrackedJob, its state.json entry is permanently stuck at running. Because state.json is an on-disk registry independent of any live process, every subsequent status/task invocation rehydrates the dead job and the scheduler continues to treat it as occupying a slot — indefinitely, with no self-healing.

Bug 3 — taskkill invoked with a shell layer by default on Windows

terminateProcessTree() (scripts/lib/process.mjs:57-98) builds the kill command as an argv array:

const result = runCommandImpl("taskkill", ["/PID", String(pid), "/T", "/F"], {
  cwd: options.cwd,
  env: options.env
});

but never passes shell: false. runCommand()'s default (scripts/lib/process.mjs:12):

shell: options.shell ?? (process.platform === "win32" ? (process.env.SHELL || true) : false)

resolves to shell: true on win32 whenever process.env.SHELL is unset (the common case for a plain Windows shell). That routes the ["/PID", "<pid>", "/T", "/F"] args through a shell instead of passing them directly to CreateProcess, which is a plausible source of arg-mangling — particularly when the process tree's ancestor is an MSYS/Git Bash shell, which is known to rewrite /-prefixed arguments (/PID → path-like rewriting) unless explicitly suppressed.

Requested fix (already speced against this exact code, not yet merged)

  1. runTrackedJob: race runner() against a timeout (env-overridable, e.g. CODEX_COMPANION_TURN_TIMEOUT_MS, default 45 min); on breach, write the job record as status: "failed" (with an explicit timeout error message) before any process exit handling runs, and clear/unref() the timer on the normal-completion path so it doesn't keep the process alive.
  2. Job-read path (listJobs or its callers): reconcile liveness before returning — for any job with status in running/queued, a numeric pid, and createdAt/startedAt older than ~15s (grace period for just-launched jobs), check the PID is actually alive; if not, rewrite it to status: "failed" in both the per-job file and the state.json registry.
  3. terminateProcessTree: pass shell: false explicitly on the taskkill call so the argv array reaches CreateProcess unmodified.

Reproduction notes

  • Bug 2 was independently reproduced by hand: append a synthetic state.json job entry with a dead numeric pid, status: "running", and a createdAt well over 15s old, then run status — with the current (unpatched) 1.0.6 code, the synthetic entry is echoed back as still running forever; there is no reconcile step to catch it.
  • Bug 1 and 3 are confirmed by direct code reading of the shipped 1.0.6 source (line references above); no timer/abort logic exists anywhere in tracked-jobs.mjs, and no call site in process.mjs overrides the win32 shell default for taskkill.
  • All three fixes were speced into a single patch (turn timeout + liveness reconcile + shell:false taskkill, touching only tracked-jobs.mjs, state.mjs, process.mjs) but a Codex-driven apply attempt against the live plugin install self-blocked on a sandbox write restriction unrelated to the bugs themselves (it could not write its diff-output file outside its task workspace root) — so as of this report the plugin is still unpatched and all three bugs are live in 1.0.6.

2026-07-11 update: we applied a local patch implementing exactly the three requested fixes to a deployed 1.0.6 install and verified it: node --check clean, synthetic dead-pid job reaped to failed on status, background-task cancel clean with no orphaned workers.

Activity

  1. axisrow commented on Jul 14, 2026

    @axisrow

    Confirming Bug 2 (PID-liveness) from an adjacent layer. I hit the same race in the broker lifecycle and the working shape is: never trust a pid until you've confirmed the endpoint it owns is live. My broker shutdown does waitForBrokerEndpoint() first, and only tree-kills by pid when the probe succeeded — otherwise an OS-reused pid gets you killing an unrelated process (scripts/lib/broker-lifecycle.mjs). The identical principle drops into listJobs: before returning a running/queued job, probe its recorded pid (process.kill(pid, 0)), and if dead, flip it to failed with a worker_died reason right there in the read path — no scheduler change needed. Bug 1 (no turn-timeout in runTrackedJob) is the write-side twin of this; fixing both makes a wedged worker self-heal on the next status. This is the root cause behind #486 and #489 too — worth treating as one surface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions