Repository navigation
Ending any Claude session kills the shared broker mid-turn: concurrent sessions' tasks die silently (exit 0) and stay "running" forever #540
Description
Environment
- Plugin:
codex@openai-codexv1.0.6 (repoopenai/codex-plugin-cc) - Claude Code on Linux (Fedora, Node 24)
- Multiple concurrent Claude Code sessions in the same workspace (interactive sessions plus occasional short-lived/headless ones)
Summary
When any Claude session in a workspace ends, the SessionEnd hook unconditionally shuts down the workspace's shared broker. If another session's rescue task is streaming a turn through that broker at that moment, its companion process exits silently with code 0 mid-turn: no error, no failed state — the job log just stops after an ordinary event line. The job file stays status: "running" forever (nothing ever pid-checks it), and the stale record then blocks all further tasks on that session ("Task … is still running") until a manual /codex:cancel.
Three defects compose into this:
-
SessionEndassumes one session per workspace.scripts/session-lifecycle-hook.mjs:101(handleSessionEnd) callssendBrokerShutdown()on the broker recorded in the shared per-workspacebroker.json, with no check whether other live sessions (or in-flight jobs) are using it. The broker'sshutdown()then callssocket.end()on every connected client (scripts/app-server-broker.mjs:104), including another session's mid-stream turn. -
A transport drop mid-turn is a silent exit, not an error.
captureTurnawaitsstate.completion(scripts/lib/codex.mjs:606), a bare promise resolved only by aturn/completednotification (resolveCompletion,lib/codex.mjs:304-310). It is never raced against the client's exit. On socket close,handleExit(scripts/lib/app-server.mjs:163) rejects only pending request promises — during streaming there are none (turn/startalready resolved) — so nothing rejects, no catch runs,runTrackedJobnever records a failure. The closed socket was the last ref'd handle (the completion timer isunref()d,lib/codex.mjs:393), so the event loop drains and Node exits 0 mid-await. The same silent death occurs for any transport loss: direct-mode app-server child crash, broker crash, or the client being SIGTERMed (no signal handler updates job state either). -
Nothing ever detects the corpse.
status/result(scripts/lib/job-control.mjs) render whatever the job file says without checking pid liveness, so the job showsrunningwith growingelapsedindefinitely. Bonus: whenresult <job-id>is called for a running/dead job,matchJobReferencethrowsNo job found for "<id>"before the intendedJob … is still runningbranch inresolveResultJobis reached (the predicate filters to finished jobs andmatchJobReferencethrows on no match), which makes diagnosing this even more confusing.
Evidence (observed corpse, twice, different sessions)
- Job
A(session 1): created 06:52:50, log ends 06:52:57 right afterCommand completed: … (exit 0); recorded pid dead; job file frozen atstatus: running,phase: verifying;statusstill showed it "running / 12m" long after. - Job
B(session 2): created 07:18:04, log ends 07:19:13 the same way. The Bash call that ran the foreground companion returned normally after ~94 s with a clean exit (no timeout, no error output beyond relayed progress lines) — i.e., the process self-exited mid-turn. - Every
/tmp/cxc-*broker session dir created during that window was erased completely (socket + pid + log + dir +broker.json) — the full-erase combination onlyhandleSessionEndproduces — while jobs were in flight; sessions were being opened/closed concurrently in this workspace throughout. - A later failure ("Selected model is at capacity") failed loudly with
Turn failed— showing the error path works when a rejection actually reaches the turn; the silent deaths never produce any error line.
Related but distinct: #526 (idle-timeout shutdown race, broker-side), #509 (stale broker reuse → hang at Turn started), #450/#380 (orphan brokers on missing/failed SessionEnd), #286 (same-cwd state races). None of them covers the SessionEnd-of-a-concurrent-session → mid-turn client kill → silent exit-0 → permanently-running job chain.
Expected behavior
- Ending one Claude session must not kill in-flight work of other sessions sharing the workspace broker (refcount sessions, or refuse/defer shutdown while a foreign turn is streaming, or scope brokers per session).
- A transport loss mid-turn must surface as a job failure (race
state.completionagainst the client's exit/close promise) — never a silent exit 0 that leavesstatus: "running". status/resultshould detect a dead worker (pid liveness check) and report the job as failed/orphaned instead ofrunningforever; a dead job must not block subsequent tasks until manual cancel.
Acceptance criteria
- With two sessions in one workspace, ending session X while session Y streams a rescue turn: Y's task either completes normally (broker survives) or fails fast with an explicit transport error — job state transitions to
failedwith an error line in the log. - Killing the companion process mid-turn (SIGTERM/SIGKILL) leaves a job that
statusreports as dead/failed after the fact, notrunning. result <job-id>for a running job says "still running", not "No job found".- Regression test: broker
shutdown()with a client mid-stream → client'scaptureTurnrejects (no silent event-loop-drain exit).
Workaround
Setting CODEX_COMPANION_APP_SERVER_ENDPOINT to an unreachable socket forces the ECONNREFUSED → direct-mode fallback in withAppServer, giving each task a private app-server that other sessions' SessionEnd cannot kill (at the cost of per-task startup).
Activity
STRML commentedon Jul 29, 2026
Ran into this on macOS today and came out with direct evidence the shared broker died mid-turn, plus a second defect that's the reason it's silent rather than just fatal.
Long session, about twelve
codex:codex-rescuereview jobs. Four hung and needed cancelling by hand; the rest finished in 1-9 minutes. Plugin v1.0.6, macOS arm64.The one with hard evidence:
task-ms52enq3-isofxkstarted at 19:44:49 local and went quiet. When I finally cancelled it, the cancel failed too:cxc-vgaxEtwasn't on disk any more.broker.jsonby then pointed at a different session dir,cxc-AtZrun, whose process started at 20:47:37, roughly 28 seconds after that failed cancel. So the broker behind the wedged job was gone and a replacement had already come up, while the client sat there getting no error and no close.The other three failed to cancel in ways that fit:
task-ms4up1vj-m5vrbt—no active turn to interrupt. Log stops dead afterTurn started, nothing for 41 minutes.task-ms4w7gvp-ftlxraandtask-ms4uitss-u3dcvm—thread not found: <threadId>, both cancelled inside the same second, both against threads the then-current broker had never heard of.I can't prove a
SessionEndhook is what killed it. I don't know that another session ended right then, and the dead broker's own log dir had already been cleaned up by the time I went looking. What I can say for certain is the broker died and nothing told the client.The second half is why that's invisible.
app-server-broker.mjsproxies every client through one sharedcodex app-serverchild and has noappClient.on("exit")handler, so nothing closes or errors the connected sockets when that child goes away. And the waiting side has no backstop:captureTurninlib/codex.mjsawaits astate.completionresolved only from the notification handler, whilehandleExitinlib/app-server.mjsrejectsthis.pendingand never touches it. Once the caller is parked on that promise, a dead upstream is indistinguishable from a slow one.Same gap #391 describes for dropped terminal events, reached from the other direction. There the events go missing; here the whole process does. Either way you're awaiting a promise with nothing left alive to settle it.
One thing that might matter for triage: two of my four stalls started with no other Codex job running at all, including the one above. A concurrent session isn't needed to get a wedge, even if it's a good way to kill the broker.
For now I watch from outside, since nothing reaps these: poll
codex-companion.mjs status --jsonand cancel any running job whoselogFilehasn't grown in five minutes. That separates cleanly, healthy jobs write a line every few seconds as they run commands, and all four stalls sat at flat zero for twenty minutes or more. Fair warning if you try the same, once the broker is dead thecancelfails too, so the local job record staysrunningand you have to clear it separately.nalarius commentedon Aug 7, 2026
Adding the piece @STRML couldn't confirm: I have both sides of the correlation on disk — the other session's end and the killed turn — plus a tested guard for defect 1.
Plugin v1.0.6, Linux. Still present on
main@ db52e28.Direct evidence that another session's
SessionEndkilled an in-flight turnTwo Claude Code sessions open on the same workspace. Session A launches a long read-only Codex task in the background; session B is unrelated and ends while A's turn is streaming.
codex-companion.mjs task --write(background)mkdir -p …/_workspacesucceeds; assistant says it will now write its notescwdThe final record in the rollout log is not
task_completebut an injected user message:{"type":"response_item","payload":{"type":"message","role":"user", "content":[{"type":"input_text","text":"<turn_aborted>\nThe user interrupted the previous turn on purpose.\n…"}]}}Nobody interrupted anything. Session A was idle-waiting; session B just exited.
Ruled out, each measured rather than assumed:
last_token_usage.input_tokens220,071 ofmodel_context_window258,400 (85%).primary.used_percent14.0,rate_limit_reached_type: null./codex:cancel—handleCancelwritesCancelled by user.to the job log. Absent.DEFAULT_STATUS_WAIT_TIMEOUT_MS) isstatus --waitonly.Control case, same workspace, same day: two more tasks launched at 15:12:32 / 15:12:33 ran ~10 and ~11 minutes — longer than the one that died — and both ended with
task_complete. No session in that workspace ended during that window. So this is not a duration or size limit.The exit code was 0 and the job record was untouched, exactly as described in the issue body:
cleanupSessionJobsfilters bysessionId, butsendBrokerShutdown/teardownBrokerSessiondo not. The broker is keyed by workspace alone (resolveBrokerStateFile→resolveStateDir(cwd)), so it is shared while its teardown is not guarded.For anyone hitting this before a fix lands: the loss is total but avoidable — a task told to write its output incrementally keeps whatever it had written, while one that buffers everything to the end loses all of it. In our case ~6 minutes of research evaporated because the write step was the last step.
Suggested guard for defect 1
Skipping teardown while another session still has a live job. Ordering matters:
cleanupSessionJobsmoves above the shutdown so the session's own jobs are reaped first, and only then is the shared broker considered.and in
handleSessionEnd:The pid-liveness check is what keeps this from becoming issue #543 — without it, a single stale
runningrecord (which this very bug produces) would pin the broker forever.Verified against a scratch git workspace with a synthetic
state.json/broker.json:running, live pidcompletedrunningrunning, dead pid (stale)This only addresses defect 1. Defects 2 and 3 are the reason it was silent rather than merely fatal, and they remain: a transport drop mid-turn still can't settle
state.completion, and nothing pid-checks a job afterwards. A guard here narrows one common trigger; racingstate.completionagainst client exit is what would make the whole class loud instead of invisible.astradevkin commentedon Aug 10, 2026
Confirming independent reproduction of this exact mechanism (plugin v1.0.6, macOS) — landed on the same root cause chain before finding this issue:
session-lifecycle-hook.mjs::handleSessionEndcallsloadBrokerSession(cwd)unconditionally and tears down the broker with no check for other sessions' in-flight jobs;rejectCompletionis constructed inlib/codex.mjs(~L304-321) but never called anywhere in the codebase, so the dying connection resolves nothing, the event loop drains, and the process exits 0 with the job frozen atstatus: "running".One data point that might sharpen the writeup: the killer session does not need to have used Codex at all. In our reproduction, the session that ended (via
/clear) had zero entries in its own job state — it had never called the companion. Its mere presence in the same workspace cwd was sufficient to triggerhandleSessionEnd→ broker teardown. Worth calling out explicitly since "avoid running Codex in multiple panes" is not a workaround — any sibling Claude Code session in the same directory is a hazard, whether or not it touches this plugin.Also worth noting for whoever picks up the fix:
cleanupSessionJobsright next to the teardown call (same file) correctly filters bysessionIdbefore acting — the asymmetry is that the broker teardown itself has no equivalent filter. The fix pattern already exists in the same file, just not applied to the broker-shutdown path.Shepherd-Creative commentedon Sep 28, 2026
Another trigger for this chain, and it needs no second session: a
claude -plaunched from inside the Codex job itself. With it, every job longer than ~20 s died on a single-session machine.Setup: plugin 1.0.6, codex-cli 0.156.1 and 0.157.0, Claude Code 2.1.281, macOS arm64.
~/.codex/config.tomlincludes a local peer-discovery MCP server (claude-peers) that runs a headless summariser when it starts, with nocwd, so it inherits the Codex task's:What happens: Codex starts that server for the task thread. The headless
claude -pis a full Claude Code session in the task's workspace, so it loads the user's Claude plugins, this one included. When it exits 17 to 22 s later, itsSessionEndrunssession-lifecycle-hook.mjs, which resolves the same state dir (same git toplevel, same Claude-sideCLAUDE_PLUGIN_DATA), finds the running job's broker inbroker.jsonand shuts it down. From there it is defect 2 of the issue body:turn_aborted(reasoninterrupted) in the rollout, the companion exits 0 and the job staysrunning.Evidence: a preload in the broker snapshotted the process table the instant
broker/shutdownarrived, which caught the sender still connected:Same run, UTC:
broker/shutdownreceived at 20:20:08.908; the broker ends its client sockets and closes the app-server's stdin at 20:20:09.001; the companion exits 0 at 20:20:09.002; the rollout recordsturn_abortedat 20:20:09.086 (duration_ms: 20648); the hook then SIGTERMs the broker pid at 20:20:09.168 (the #743 path). It happened in 4 of 4 runs: two real jobs and two reproductions on a trivial read-only task, each 17 to 22 s after the MCP server started.Why this shape is worth having on record
codex:codex-rescuethe relayed "result" was that preamble. Throughadversarial-reviewit was the schema-shaped commentary message,{"verdict":"needs-attention","summary":"I'm tracing ...","findings":[],"next_steps":[]}, which reads as a real verdict with no findings.claude -pin the workspace (an MCP server, a hook script, a summariser) counts, including a child of the Codex job itself.~/.codex/plugins/cache/openai-codex/codex/1.0.6/). Those copies ran SessionStart and SessionEnd in every one of my runs but resolved a different state dir, because Codex setsCLAUDE_PLUGIN_DATAto~/.codex/plugins/data/codex-openai-codex. Harmless here, but a fix that changes how the state dir is resolved could let them reach the same broker.Workaround for this trigger: stop the child session starting. For claude-peers that is
CLAUDE_PEERS_V2_SUMMARY_MODE = "disabled"under[mcp_servers.<name>.env]; with it, a job forced to 67 s completed normally. The checks proposed above and in #612 would cover it too, since the headless session has its ownsession_idand no jobs of its own.minhoyooDEV commentedon Sep 29, 2026
Adding frequency data and a PR-triage note from macOS (plugin 1.0.6, codex-cli 0.156.1, Claude Code 2.1.284, Darwin 25.6 arm64).
How often: of 83 Codex threads started from Claude Code in September, 3 ended in
turn_aborted/interrupted, all in the one workspace where several Claude sessions run side by side. 2 of the 3 line up with another session ending:/clearat 12:03:36.4 (0.8 s later)Downstream cost: the waiting session kept polling
statuson the orphaned job (dead pid,running) for ~27 minutes, until the job was cancelled by hand.Which open PR covers this case: the killed jobs were foreground
codex:codex-rescuetasks of a different session.background: truejobs.b2b14fcpasses 144/144 on macOS arm64 / Node 26.8.1 (maindb52e28: 91/91). Run the suite withCLAUDE_PLUGIN_DATAandCODEX_COMPANION_SESSION_IDunset; inside a live Claude Code session 4 tests fail on main too (fix(tests): keep the suite hermetic inside a live Claude Code session #456).tosoyn commentedon Sep 29, 2026
Another independent confirmation (plugin 1.0.6, Linux, Claude Code with four profiles and several concurrent sessions per workspace), plus a minimal isolated repro that needs no Codex at all.
Repro, no real broker needed. In a temp git workspace with a temp
CLAUDE_PLUGIN_DATA, writebroker.jsonfor that workspace pointing at a stand-in process that leads its own process group (setsid sleep 300), just like a real detached broker. Then run the hook as a different session:The stand-in dies (
kill(-pid, SIGTERM)fromterminateProcessTree) andbroker.jsonis removed, although the ending session never used that broker. Control run withoutbroker.json: the process survives.SessionEndalso fires on/clear, so clearing one session is enough to kill a neighbour's in-flight job.The other side: leaked brokers. When a session is killed without
SessionEnd(terminal closed), its broker stays forever. On this machine three brokers from 2026-09-23 were still alive 5.5 days later, each a process group of 12–17 processes (broker,codex app-server, and every MCP server from~/.codex/config.toml: qmd ×2, chrome-devtools-mcp, runpod-mcp, node_repl, a Python MCP), about 528 MB RSS together. All their jobs had finished on 09-23 and no Claude session had those directories as cwd.Both sides point to the same fix: the broker needs an owner count (or per-session leases) and an idle timeout, and
SessionEndshould only shut it down when no other session holds a lease.rhristow commentedon Oct 7, 2026
Still happens on plugin 1.0.6 with Claude Code 2.1.292 (Linux). Deterministic repro: in a git repo, start a
codex-task --effort lowthat runssleep 45, then runclaude -p "Reply with ok."in the same directory. The task exits 0 mid-command with no answer and its job stays "running". With--settings '{"enabledPlugins":{"codex@openai-codex":false}}'on the second session, the task completes. Open fixes that would close it: #623, #491.LukeSal88 commentedon Oct 8, 2026
Another independent reproduction with exact timing (plugin 1.0.6, Claude Code 2.1.293, macOS, 2026-10-08).
In a scratch git worktree with no other Claude Code session in it, a
task --backgroundwasrunning. A throwaway
claude -p "…"then started and exited in the same directory. The task'srollout recorded
turn_aborted(reasoninterrupted) at 06:18:57.198Z, the second thatclaude -pexited, andstatus <job>kept reportingrunninguntil we cancelled the job.A workaround we use until #491 or a similar fix lands, in case it helps others: after each
status <job> --waitcycle, read the last record of~/.codex/sessions/**/rollout-*-<threadId>.jsonl. If it isturn_abortedand itsturn_idequals the job's
turnId,cancelthe job and resume the thread once withtask --resume-last --background. Matching the turn id stops an earlier abort on the samethread from counting against the resumed job.
tatsumiyama commentedon Oct 15, 2026
We hit the same thing on Windows 11 with plugin 1.0.1 (two sessions in one working directory). Two details that may help:
Suggestions in addition to reference counting: refuse or defer broker/shutdown while a request is active, bound shutdown with a socket destroy and a timer, set closed in handleExit, and health-check the broker with a lightweight RPC.