CLI worker silent death after parallel-command burst (announce → mkdir → probe → exit) on Linux/WSL2 — Turn completed event never fires #21937

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   app-serverIssues involving app server protocol or interfacesIssues involving app server protocol or interfaces

   tool-callsIssues related to tool callingIssues related to tool calling

   windows-osIssues related to Codex on Windows systemsIssues related to Codex on Windows systems

   on May 9, 2026
2. henkterharmsel commented on May 12, 2026

   

   henkterharmsel

   on May 12, 2026

   Last edited by henkterharmsel

   More actions

   Confirming this reproduces on plain Linux (no WSL). Wrapper is the OpenClaw gateway spawning codex app-server --listen stdio:// as a child via @openai/codex plugin runtime.

   Same outcome as the OP — worker stops emitting mid-turn, no task_complete event, no error, no SIGTERM. The rollout JSONL file just ends. From the wrapper side, the completion promise hangs indefinitely; only an external abort frees the lane.

   Last events in the rollout before silence (from $CODEX_HOME/sessions/2026/05/11/rollout-*.jsonl, redacted):

   13:36:53.024Z  event_msg      payload=agent_message       <announcement message>
   13:36:53.034-063Z  response_item  payload=function_call × 4  ← parallel batch (psql/sed/shell mix)
   13:36:53.076Z  event_msg      payload=token_count
   13:36:54.714-55.171Z  event_msg  payload=exec_command_end × 4   ← all exit 0
   13:36:56.718-19Z   response_item  payload=function_call_output × 4
   13:37:03.424Z  response_item     payload=reasoning         ← LAST event written, then silence
                                                                 no task_complete, no error event, no SIGTERM
   

   So the OP's symptom isn't WSL-specific; it reproduces on a vanilla Linux host too. Repro rate is high enough that we accumulate ~6 hung session records per active day across telegram-channel-driven dispatch, all with the same shape: announcement → parallel batch → reasoning → silence. Also previously seen on codex-cli 0.128.0 in the same wrapper, so not introduced by 0.130.

   I scanned the 60 commits in rust-v0.130.0..rust-v0.131.0-alpha.6 and didn't spot one that looks like it would fix this (closest near-misses: #21825 transport timeout bump, #21778 turn-notification routing by ID — neither addresses the silent worker exit specifically).

   Happy to share full rollout fragments or wrapper-side stderr/diagnostic logs if that helps narrow it down.
3. gupta-ash commented on May 19, 2026

   

   gupta-ash

   on May 19, 2026

   More actions

   Adding two independent reproductions of this on Windows 11 + Codex CLI 0.129.0 in the past 48 hours. Same downstream symptom (no Turn completed notification; companion's state.completion Promise hangs indefinitely; outer Bash timeout is the only bound).

   Reproduction 1 — Windows 11 + Obsidian vault content review (2026-05-17)

   • CLI version: 0.129.0 (via claude_codex_plugin-cc plugin v1.0.4 baseline)
   • Workspace cwd: D:/Obsidian/KidsObs (non-git directory)
   • Transport: broker mode (companion connected to long-lived broker via Windows Named Pipe)
   • Sandbox: read-only (per task mode default)
   • Trigger: ~7 KB prompt with 16 cited file paths; discriminator was instructed to spot-check ~30 PowerShell Get-Content + Select-String commands against vault chapter files for evidence-tag accuracy
   • Active work window: 4 min 41 sec (01:10:31 → 01:15:12 Z); final events were 3 simultaneous Command failed: pwsh ... Get-Content (exit -1) within 44 ms
   • Stall window: silent thereafter until Bash 10-min outer timeout killed the wrapper
   • Tracked-job log endpoint: ${TMPDIR}/codex-companion/${slug}-${hash}/jobs/task-mpaic5td-xu0oqz.log, last entry Command failed ... (exit -1), zero further activity
   • state.json snapshot: status: "running", pid: 38644, updatedAt: 2026-05-18T01:11:39Z (stale — updatedAt only refreshes on phase/threadId/turnId changes, not on every progress event; log file is the truth)
   • Subsequent runs of similar prompt structure: completed successfully (one took 20 min). Confirms non-determinism.

   Reproduction 2 — Windows 11 + neev workspace content review (2026-05-18, 18 hours later)

   • CLI version: 0.129.0
   • Workspace cwd: E:/dev/neev (git repo)
   • Transport: broker mode
   • Sandbox: read-only
   • Trigger: Phase C iter-1 cross-family discriminator review on financial-strategy.md
   • Active work window: ~3 min 42 sec (19:03:47 → 19:07:29 Z); rapid alternation of findstr, Get-Content, Select-String, and some git diff --no-index commands. A few Command declined (exit -1) interleaved with successful completions before the wedge
   • Stall observed by orchestrator: 31+ minutes of "running" status with no further log activity before manual cancel attempt
   • Tokens consumed before wedge: 1,026,771 input tokens (979K cached) per orchestrator's diagnostic — possibly relevant if there's a context-window or rate-limit edge
   • Tracked-job log endpoint: ${TMPDIR}/codex-companion/neev-afc8637dfd450973/jobs/task-mpbkoe5v-odz1t1.log, 68 lines preserved
   • Codex session JSONL endpoint: ~/.codex/sessions/2026/05/18/rollout-2026-05-18T12-04-01-019e3c79-4725-7f43-9ddc-12b391fb4879.jsonl, 144 lines; last entry function_call_output at 2026-05-18T19:07:29.903Z
   • Codex worker pid 8624: GONE post-stall (presumably exited naturally; no kill signal sent before)
   • Codex CLI health post-stall: healthy — trivial codex exec 'echo ok' returned in 3.3s. Bug is session-specific, not codex-down

   Pattern observation across all three reports (yours + ours × 2)

   Dim	@vstandos (Linux/WSL2)	Ours Rep1 (Win11/KidsObs)	Ours Rep2 (Win11/neev)
   OS	Linux/WSL2	Windows 11	Windows 11
   CLI version	0.129.0 / 0.130.0	0.129.0	0.129.0
   Sandbox	danger-full-access	read-only	read-only
   Active span before wedge	(varied)	4m 41s	3m 42s
   Trigger shape	"psql/sed parallel batch → assistant msg → mkdir → 1 more probe → silent exit"	parallel Get-Content batch with some exit -1 failures	rapid findstr/Get-Content/git diff alternation
   Common downstream	no turn.completed, worker exits 0, state.completion hangs	same	same

   Whatever the proximate trigger (heavy parallel tool-call activity; failures in the batch; possible context-window threshold), the failure mode is consistent: codex CLI app-server worker exits without emitting turn/completed. The wrapper layer (companion) has no internal timeout on state.completion and waits forever.

   Companion-layer workaround we shipped

   For our environment (Windows + Claude Code orchestrating discriminator reviews against codex), we built a thin Node wrapper that sits between the orchestrator and codex-companion.mjs. The wrapper:

   1. Spawns codex-companion.mjs as a subprocess (transparent argv pass-through).
   2. Reads the broker session file (${stateDir}/broker.json) for the broker pid.
   3. Periodically enumerates the broker's full process-tree descendants via wmic process where (ParentProcessId=<broker>) get ProcessId,ParentProcessId,Name /format:csv — note: codex.exe is a grandchild of broker under Windows shell-true spawning (broker → cmd.exe → codex.exe), NOT a direct child, so direct-child enumeration would miss it.
   4. Filters /codex/i-named descendants. If a previously-tracked descendant disappears (verified via process.kill(pid, 0) returning ESRCH) AND the companion subprocess is still alive AND the zero-descendant state persists for 2 consecutive polls (debounce against legitimate process replacement) → trip.
   5. On trip: taskkill /T /F the companion tree + emit a structured JSON envelope marking the run as verdict: "tooling-failure" (downstream orchestrator already handles this verdict).
   6. Backstop: 15-min log-file mtime stagnation check filters jobs by CODEX_COMPANION_SESSION_ID env var (passed to child) to avoid cross-wrapper log false-positives.
   7. Stdout single-write contract preserved (only ONE of: companion's clean output OR tooling-failure envelope).

   Detection latency: ~2-4 sec (process watchdog) or 15 min (time fallback). Empirically validates against the smoke 6 + smoke 7 scenarios we documented.

   This is a wrapper, not an upstream patch — we don't maintain a fork of codex-companion.mjs. The wrapper imports stdlib only (no junction-protected imports) and works by observing the broker's process tree externally. Source on request if upstream finds it useful.

   Secondary issue spotted: codex-companion.mjs cancel fails to kill the wedged tree under Git-Bash on Windows

   When trying to cancel reproduction-2 manually, codex-companion.mjs cancel <jobid> invoked from Git-Bash on Windows produced:

   taskkill /PID 8624 /T /F: exit=1: ERROR: Invalid argument/option - 'E:/Prg/Git/PID'.
   Type "TASKKILL /?" for usage.
   

   Root cause: lib/process.mjs terminateProcessTree() uses spawnSync("taskkill", [...], { shell: process.env.SHELL || true }). With process.env.SHELL = "/usr/bin/bash" (Git-Bash), bash's MSYS path-conversion rule rewrites the leading-slash arg /PID → E:/Prg/Git/PID (or similar Git install path) before taskkill sees it.

   Workaround that worked: invoke taskkill via child_process.spawnSync WITHOUT shell: true (Node's child_process resolves taskkill on PATH directly without going through bash; CreateProcessW gets the args as-is). Our wrapper does this; codex-companion's own cancel path does not. Worth a small fix.

   Suggested upstream patches

   Two independent surfaces:

   A. state.completion race / silent worker exit:

   • enable structured logging on app-server worker exit path (record what the model was doing right before exit; was there a pending API call? a stalled tool result?)
   • emit a worker-died-mid-turn event so wrapper layers can react instead of hang indefinitely
   • consider exposing an idle-progress timeout config on the turn (e.g., --turn-no-progress-timeout 600000) — would let companions opt-in to a sane upper bound

   B. lib/process.mjs Git-Bash taskkill:

   • pass shell: false (or omit shell on Windows entirely) when invoking taskkill /PID. The arg pattern matters less than avoiding bash arg-mangling.

   Happy to share more reproduction artifacts (log files, JSONL session dumps, our wrapper source) if useful.
4. bondtesty commented on May 19, 2026

   

   bondtesty

   on May 19, 2026

   More actions

   Adding another downstream reproduction from a Linux app-server wrapper/integration. I am keeping project paths, prompts, thread contents, and agent identifiers out of this public report.

   Environment

   • Codex CLI: 0.130.0
   • OS: Ubuntu 22.04 / x86_64
   • Transport: codex app-server --listen stdio://
   • Sandbox: danger-full-access
   • Approval policy: never
   • Auth/provider: ChatGPT-authenticated Codex app-server path
   • Integration shape: a long-lived local wrapper starts app-server over stdio, sends initialize, thread/resume, then turn/start, and waits for turn/completed before marking the runtime wake complete.

   Observed failure

   A resumed app-server turn produced partial output and completed tool executions, then never emitted a terminal turn event. The wrapper remained waiting for turn/completed until it was externally diagnosed/restarted.

   Sanitized raw event shape:

   turn/start result: turn.status = inProgress
   turn/started
   item/agentMessage/delta ...
   item/completed: assistant progress message
   thread/tokenUsage/updated
   item/started: commandExecution A
   item/started: commandExecution B
   item/commandExecution/outputDelta: command A output
   item/commandExecution/outputDelta: command B output
   item/completed: commandExecution A, exitCode 0
   item/completed: commandExecution B, exitCode 0
   <stream stops here>
   

   Local artifact counts from the stalled run:

   stdout jsonl: 70 lines
   normalized event jsonl: 57 lines
   stderr: 0 bytes
   raw grep turn/completed: 0 matches
   normalized grep turn_end|error|timeout: 0 matches
   last normalized event: tool_output
   

   There was no turn/completed, no top-level error, no warning, no stderr output, and no non-zero command exit code in the stalled turn.

   Why this looks related

   This matches the downstream failure mode described here: the app-server turn has clearly accepted and progressed, command tools have returned, but the completion notification never arrives, so the wrapper's completion wait has no terminal signal.

   It is not the apply_patch duplicate-approval case (#21117) in this reproduction: the tools were ordinary shell command executions, not apply_patch.

   Recovery / workaround

   Restarting the app-server-backed runtime and re-requesting the same logical work recovered immediately. The follow-up turn completed normally.

   Our local mitigation will be to bound the app-server terminal-event wait and fail explicitly with a diagnostic like "app-server timed out waiting for turn/completed after tool output" rather than letting the collaborator lane wait indefinitely. That does not fix the upstream issue, but it makes the failure recoverable and easier to classify.

   Happy to provide additional redacted event tails if maintainers want a specific shape, but I cannot post full prompts or session transcripts publicly.
5. srosro

   added a commit that references this issue on May 20, 2026

   feat(pipeline): codex hang watchdog + one retry (#86)

   ...

   07db9f2
6. srosro

   added a commit that references this issue on May 29, 2026

   feat(observability): surface codex watchdog kills + worker-attribute …

   ...

   fec15f6
7. chenditc commented on Jun 13, 2026

   

   chenditc

   on Jun 13, 2026

   More actions

   Adding a reliably-observed, related repro of the "turn never completes / Turn completed never fires" failure, in case it helps. Seen on codex-cli 0.136.0 driving codex app-server --stdio in goal mode — looks like the permanent (non-recovering) variant of this issue, closely related to #23807 (same ~300s gap value) and the active-turn-control meta-bug #25779.

   Setup

   • codex app-server --stdio, codex-cli 0.136.0
   • a model provider with wire_api = "responses", base_url = an OpenAI-compatible /v1/responses endpoint
   • one thread/goal/set { status: "active" } + a single turn/start; the client then only reads notifications and auto-approves exec requests (never sends another turn)
   • model: gpt-5.x family, high reasoning effort

   What happens

   The model backgrounds a long-running shell command and then polls it with an empty write_stdin:

   function_call exec_command {"cmd":"<long-running job>"}  -> "... Process running with session ID N"
   function_call write_stdin  {"session_id":N,"chars":"","yield_time_ms":900000}
   

   Each empty poll returns after exactly ~300.0 s — the background-terminal ceiling (DEFAULT_MAX_BACKGROUND_TERMINAL_TIMEOUT_MS = 300_000; an empty write_stdin is clamped to max_write_stdin_yield_time_ms, not the 30 s cap) — with the process still running. codex records the function_call_output + token_count. The second-to-last such poll correctly re-prompts the model; after the last one, codex never issues the next /v1/responses request. No turn/completed, no error; thread/goal/updated stays status:"active" forever.

   Forensics (from the rollout JSONL)

   • Payload-type census for the wedged turn: 1x task_started, 0x turn_complete / error / abort. function_call and function_call_output counts are equal (e.g. 28 ↔ 28) — every tool call, including the final poll, got a result. No dangling tool call; the loop just stops after a successful tool result. One turn id throughout — it never completes.
   • The /v1/responses traffic for that run was 28/28 = HTTP 200 + data: [DONE], zero failures / zero retries — so this is not a model-call hang or a rate-limit; the next request is simply never sent.

   Why goal continuation can't recover it

   In core/src/goals.rs, maybe_continue_goal_if_idle / maybe_start_goal_continuation_turn bail whenever active_turn.is_some(). The block that emits TurnComplete, clears active_turn, and fires MaybeContinueIfIdle (core/src/tasks/mod.rs) only runs when the model returns a tool-free response. Because the model keeps ending each step with another write_stdin poll, that block is never reached, active_turn stays Some, and continuation is structurally blocked — so one wedged turn-step silently kills the whole goal.

   Minimal repro

   codex app-server --stdio        # 0.136.0; provider wire_api="responses" -> any /v1/responses endpoint
   -> initialize
   -> thread/new
   -> thread/goal/set { threadId, goal:{ objective:"Keep working until the background job finishes, then stop.", status:"active" } }
   -> turn/start     { threadId, input:"Run `bash -c 'for i in $(seq 1 60); do echo working $i; sleep 10; done'` in the background, then poll it with write_stdin until it exits, then report done." }
   # client: read notifications only, auto-approve every exec request, never send another turn.
   

   The model backgrounds the job and polls via empty write_stdin (300 s ceilings). After a poll that returns while the job is still running, no further /v1/responses is issued and no turn/completed ever fires.

   Note: intermittent — it only wedges on a poll that returns with the process still running; polls that return after the process exits always continue normally. May take several poll cycles to hit.

   Expected

   Either (a) after an exec/write_stdin poll result the turn-step loop reliably issues the next model request and eventually emits turn/completed; or (b) if a between-turn-step dispatch stalls (cf. #23807), it surfaces a visible error/retry and the turn fails loudly instead of leaving the session — and any active goal — silently stuck active. In app-server goal mode specifically, a turn that cannot advance should fail the turn (clearing active_turn) so maybe_continue_goal_if_idle can re-drive the goal.
8. waiting761 commented on Jul 4, 2026

   

   waiting761

   on Jul 4, 2026

   More actions

   Adding another independent reproduction of this exact failure mode — Windows 11, via the Claude Code plugin's codex-companion.mjs → broker → app-server path (same architecture @gupta-ash described above: broker → cmd.exe → codex.exe as grandchild, not direct child). This adds a concrete new data point: direct process-level evidence that the stuck-turn's CPU is going into the Windows sandbox-setup helper itself, plus a negative-reproduction result.

   Environment

   • codex-cli 0.142.2, Windows 11
   • sandbox_mode = "workspace-write", [sandbox_workspace_write] network_access = true (not read-only — noting this because [Windows] read-only sandbox mode causes MCP shell_command to stall indefinitely — no function_call_output returned #13987 reports a similar stall but scoped specifically to read-only mode; this report is workspace-write)
   • Transport: Claude Code plugin → codex-companion.mjs → broker (named pipe) → cmd.exe /d /s /c "codex app-server" → codex.js app-server (node) → codex.exe (native win32-x64 vendor binary)

   What happened (live incident, captured mid-hang)

   A turn issued 3 concurrent shell_command calls (each a simple local-file Get-Content, timeout_ms: 10000), 29ms apart. All three were logged as "Running command" by the companion. From that point on, no further events were ever written — no function_call_output, no task_complete, nothing — for 32+ minutes at time of inspection, with the job still reporting status: "running".

   Live process-tree inspection at the 32-minute mark showed:

   codex.exe (native, executing the turn)
    ├ powershell.exe (the shell wrapper for one Get-Content call) — Responding=True, CPU=2.8s (not the bottleneck)
    ├ codex-windows-sandbox-setup.exe  started ~9s after dispatch, CPU=1035s
    ├ codex-windows-sandbox-setup.exe  started ~11s after dispatch, CPU=977s
    └ codex-windows-sandbox-setup.exe  started ~11s after dispatch, CPU=972s
   

   All three codex-windows-sandbox-setup.exe processes (one per concurrent shell_command) were Responding=True (not deadlocked — actively spinning/busy) and had each burned 970-1035 CPU-seconds without ever handing a result back to codex.exe. This is consistent with the "worker exits or hangs mid-turn, no turn/completed, wrapper has no timeout" pattern already documented in this thread, but pinpoints the CPU activity specifically to the Windows sandbox-setup helper process rather than the app-server/worker process itself.

   Separately, we have one historical job (different session, months back) that hung in the same way for 24+ hours until it was ended passively by codex app-server connection closed. — i.e., the only thing that ever terminated it was the transport dying, not any turn-level timeout.

   Negative reproduction (important caveat)

   We tried to build a minimal repro: same architecture, 3 concurrent Get-Content calls against small scratch files (vs. the original incident's larger workspace and deeper file path), with an even tighter concurrent-dispatch window (1ms / 7ms apart vs. 29ms in the live incident). Both attempts completed normally in 53-60 seconds, with 0 hangs. So "3 concurrent shell_commands" alone is not a sufficient trigger — this matches @gupta-ash's "non-determinism" observation above. Candidate differentiating factors we have not yet verified: workspace size/depth, file path depth, system-wide process state at time of dispatch (concurrent codex instances, other CUA/runtime processes active), or a cold-start race in the sandbox helper's setup path.

   No timeout at any layer

   Confirmed by reading the wrapper source: the companion layer has no turn-level/idle-progress timeout — only a manual cancel path exists (which does have a working terminateProcessTree, it's just never invoked automatically). This matches the fix direction already proposed in this thread (idle-progress timeout / worker-died-mid-turn event).

   Workaround in place locally

   We've added a local watchdog external to the wrapper that monitors the process tree and force-kills after an idle period with no log progress, to bound the damage — this is a local mitigation only, not a fix for the underlying hang.

   Happy to share the raw job state file / rollout JSONL timestamps (redacted) if useful for triage.
9. argszero commented on Aug 26, 2026

   

   argszero

   on Aug 26, 2026

   More actions

   This issue and #40010 are the same failure class, and I traced the exit mechanism on the macOS side — the observations here match it exactly.

   Root cause (from #40010, verified against source)

   A clean exit 0 mid-turn from codex app-server is only possible through the ConnectionClosed break in the single-client stdio path (app-server/src/lib.rs:1036–1038: single-client stdio breaks immediately instead of draining). The stdio reader thread only ends on EOF or a read error — and since nothing in the app-server calls process::exit, the EOF must come from the wrapper side closing its write end of the pipe.

   That maps 1:1 onto every reproduction in this thread:

   • "worker stops emitting mid-turn, no task_complete event, no error, no SIGTERM, rollout JSONL just ends" (henkterharmsel) — exactly what a clean ConnectionClosed break produces
   • All four wrappers (OpenClaw/OpenClaw gateway, Obsidian companion, @openai/codex plugin via codex-companion.mjs, Claude Code plugin) talk to app-server over stdio and are positioned to hit EOF on the server's stdin when the wrapper-side write end closes for any client-side reason
   • "worker process exits with code 0" — the process doesn't crash; it takes the clean path

   Why it's invisible

   The processor exit path logs an unconditional info! (lib.rs:1191–1196) with no turn-state context, so a mid-turn exit looks identical to a healthy shutdown in the logs. That's why none of the repros produced a diagnostic.

   Proposed fix (design complete, implementation pending)

   Change that unconditional exit info! to a conditional warn! when running_turn_count > 0 at processor exit, including the connection count and shutdown-forced state in the log line. Next occurrence of this class then leaves an explicit "processor task exited while a turn was still running" breadcrumb instead of silence. This covers both #40010 (macOS) and this issue (Linux/WSL2/Windows wrappers) since they share the same exit path.

   Full design discussion is in #40010 (issuecomment-5403178963 for the root-cause trace, issuecomment-5412055098 for the fix shape). I've prepared the patch but can't open a PR yet — the repo's contribution process is invitation-gated, so I'm waiting on maintainer sign-off there. Happy to post it as soon as that's cleared, or hand the diff to a maintainer who wants it faster.

   One caveat: chenditc's reproduction (2026-06-13) mentions a "permanent (non-recovering) variant" with a ~300s gap tied to #23807 — that may be a different mechanism (active-turn coordination stuck rather than process exit). The analysis above covers the process-exit variant; if the permanent variant still reproduces after the warn-log lands, the new breadcrumb will disambiguate it.
10. kv100

    added a commit that references this issue on Sep 11, 2026

    merge: codex turn races worker exit and retries once on every path

    ...

    ead7ab9