Skip to content

Ending any Claude session kills the shared broker mid-turn: concurrent sessions' tasks die silently (exit 0) and stay "running" forever #540

Open

Description

@leitxio

Environment

  • Plugin: codex@openai-codex v1.0.6 (repo openai/codex-plugin-cc)
  • Claude Code on Linux (Fedora, Node 24)
  • Multiple concurrent Claude Code sessions in the same workspace (interactive sessions plus occasional short-lived/headless ones)

Summary

When any Claude session in a workspace ends, the SessionEnd hook unconditionally shuts down the workspace's shared broker. If another session's rescue task is streaming a turn through that broker at that moment, its companion process exits silently with code 0 mid-turn: no error, no failed state — the job log just stops after an ordinary event line. The job file stays status: "running" forever (nothing ever pid-checks it), and the stale record then blocks all further tasks on that session ("Task … is still running") until a manual /codex:cancel.

Three defects compose into this:

  1. SessionEnd assumes one session per workspace. scripts/session-lifecycle-hook.mjs:101 (handleSessionEnd) calls sendBrokerShutdown() on the broker recorded in the shared per-workspace broker.json, with no check whether other live sessions (or in-flight jobs) are using it. The broker's shutdown() then calls socket.end() on every connected client (scripts/app-server-broker.mjs:104), including another session's mid-stream turn.

  2. A transport drop mid-turn is a silent exit, not an error. captureTurn awaits state.completion (scripts/lib/codex.mjs:606), a bare promise resolved only by a turn/completed notification (resolveCompletion, lib/codex.mjs:304-310). It is never raced against the client's exit. On socket close, handleExit (scripts/lib/app-server.mjs:163) rejects only pending request promises — during streaming there are none (turn/start already resolved) — so nothing rejects, no catch runs, runTrackedJob never records a failure. The closed socket was the last ref'd handle (the completion timer is unref()d, lib/codex.mjs:393), so the event loop drains and Node exits 0 mid-await. The same silent death occurs for any transport loss: direct-mode app-server child crash, broker crash, or the client being SIGTERMed (no signal handler updates job state either).

  3. Nothing ever detects the corpse. status/result (scripts/lib/job-control.mjs) render whatever the job file says without checking pid liveness, so the job shows running with growing elapsed indefinitely. Bonus: when result <job-id> is called for a running/dead job, matchJobReference throws No job found for "<id>" before the intended Job … is still running branch in resolveResultJob is reached (the predicate filters to finished jobs and matchJobReference throws on no match), which makes diagnosing this even more confusing.

Evidence (observed corpse, twice, different sessions)

  • Job A (session 1): created 06:52:50, log ends 06:52:57 right after Command completed: … (exit 0); recorded pid dead; job file frozen at status: running, phase: verifying; status still showed it "running / 12m" long after.
  • Job B (session 2): created 07:18:04, log ends 07:19:13 the same way. The Bash call that ran the foreground companion returned normally after ~94 s with a clean exit (no timeout, no error output beyond relayed progress lines) — i.e., the process self-exited mid-turn.
  • Every /tmp/cxc-* broker session dir created during that window was erased completely (socket + pid + log + dir + broker.json) — the full-erase combination only handleSessionEnd produces — while jobs were in flight; sessions were being opened/closed concurrently in this workspace throughout.
  • A later failure ("Selected model is at capacity") failed loudly with Turn failed — showing the error path works when a rejection actually reaches the turn; the silent deaths never produce any error line.

Related but distinct: #526 (idle-timeout shutdown race, broker-side), #509 (stale broker reuse → hang at Turn started), #450/#380 (orphan brokers on missing/failed SessionEnd), #286 (same-cwd state races). None of them covers the SessionEnd-of-a-concurrent-session → mid-turn client kill → silent exit-0 → permanently-running job chain.

Expected behavior

  • Ending one Claude session must not kill in-flight work of other sessions sharing the workspace broker (refcount sessions, or refuse/defer shutdown while a foreign turn is streaming, or scope brokers per session).
  • A transport loss mid-turn must surface as a job failure (race state.completion against the client's exit/close promise) — never a silent exit 0 that leaves status: "running".
  • status/result should detect a dead worker (pid liveness check) and report the job as failed/orphaned instead of running forever; a dead job must not block subsequent tasks until manual cancel.

Acceptance criteria

  • With two sessions in one workspace, ending session X while session Y streams a rescue turn: Y's task either completes normally (broker survives) or fails fast with an explicit transport error — job state transitions to failed with an error line in the log.
  • Killing the companion process mid-turn (SIGTERM/SIGKILL) leaves a job that status reports as dead/failed after the fact, not running.
  • result <job-id> for a running job says "still running", not "No job found".
  • Regression test: broker shutdown() with a client mid-stream → client's captureTurn rejects (no silent event-loop-drain exit).

Workaround

Setting CODEX_COMPANION_APP_SERVER_ENDPOINT to an unreachable socket forces the ECONNREFUSED → direct-mode fallback in withAppServer, giving each task a private app-server that other sessions' SessionEnd cannot kill (at the cost of per-task startup).

Activity

  1. STRML commented on Jul 29, 2026

    @STRML

    Ran into this on macOS today and came out with direct evidence the shared broker died mid-turn, plus a second defect that's the reason it's silent rather than just fatal.

    Long session, about twelve codex:codex-rescue review jobs. Four hung and needed cancelling by hand; the rest finished in 1-9 minutes. Plugin v1.0.6, macOS arm64.

    The one with hard evidence: task-ms52enq3-isofxk started at 19:44:49 local and went quiet. When I finally cancelled it, the cancel failed too:

    Codex turn interrupt failed: connect ENOENT /var/folders/c2/.../T/cxc-vgaxEt/broker.sock
    

    cxc-vgaxEt wasn't on disk any more. broker.json by then pointed at a different session dir, cxc-AtZrun, whose process started at 20:47:37, roughly 28 seconds after that failed cancel. So the broker behind the wedged job was gone and a replacement had already come up, while the client sat there getting no error and no close.

    The other three failed to cancel in ways that fit:

    • task-ms4up1vj-m5vrbt — no active turn to interrupt. Log stops dead after Turn started, nothing for 41 minutes.
    • task-ms4w7gvp-ftlxra and task-ms4uitss-u3dcvm — thread not found: <threadId>, both cancelled inside the same second, both against threads the then-current broker had never heard of.

    I can't prove a SessionEnd hook is what killed it. I don't know that another session ended right then, and the dead broker's own log dir had already been cleaned up by the time I went looking. What I can say for certain is the broker died and nothing told the client.

    The second half is why that's invisible. app-server-broker.mjs proxies every client through one shared codex app-server child and has no appClient.on("exit") handler, so nothing closes or errors the connected sockets when that child goes away. And the waiting side has no backstop: captureTurn in lib/codex.mjs awaits a state.completion resolved only from the notification handler, while handleExit in lib/app-server.mjs rejects this.pending and never touches it. Once the caller is parked on that promise, a dead upstream is indistinguishable from a slow one.

    Same gap #391 describes for dropped terminal events, reached from the other direction. There the events go missing; here the whole process does. Either way you're awaiting a promise with nothing left alive to settle it.

    One thing that might matter for triage: two of my four stalls started with no other Codex job running at all, including the one above. A concurrent session isn't needed to get a wedge, even if it's a good way to kill the broker.

    For now I watch from outside, since nothing reaps these: poll codex-companion.mjs status --json and cancel any running job whose logFile hasn't grown in five minutes. That separates cleanly, healthy jobs write a line every few seconds as they run commands, and all four stalls sat at flat zero for twenty minutes or more. Fair warning if you try the same, once the broker is dead the cancel fails too, so the local job record stays running and you have to clear it separately.

  2. nalarius commented on Aug 7, 2026

    @nalarius

    Adding the piece @STRML couldn't confirm: I have both sides of the correlation on disk — the other session's end and the killed turn — plus a tested guard for defect 1.

    Plugin v1.0.6, Linux. Still present on main @ db52e28.

    Direct evidence that another session's SessionEnd killed an in-flight turn

    Two Claude Code sessions open on the same workspace. Session A launches a long read-only Codex task in the background; session B is unrelated and ends while A's turn is streaming.

    Time (UTC) Event Source
    15:03:57 Session A launches codex-companion.mjs task --write (background) A's transcript
    15:08:22 Last model activity — 76 tool calls done, research complete rollout log
    15:08:25 mkdir -p …/_workspace succeeds; assistant says it will now write its notes rollout log
    15:09:52 Session B's last activity — same cwd B's transcript
    15:10:18 Turn aborted, 26s later rollout log

    The final record in the rollout log is not task_complete but an injected user message:

    {"type":"response_item","payload":{"type":"message","role":"user",
     "content":[{"type":"input_text","text":"<turn_aborted>\nThe user interrupted the previous turn on purpose.\n…"}]}}

    Nobody interrupted anything. Session A was idle-waiting; session B just exited.

    Ruled out, each measured rather than assumed:

    • Context — last_token_usage.input_tokens 220,071 of model_context_window 258,400 (85%).
    • Rate limit — primary.used_percent 14.0, rate_limit_reached_type: null.
    • Client timeout — the wrapping call had a 600 s timeout; the process died at 384 s.
    • /codex:cancel — handleCancel writes Cancelled by user. to the job log. Absent.
    • A companion-side watchdog — there is none; the only timeout constant (DEFAULT_STATUS_WAIT_TIMEOUT_MS) is status --wait only.

    Control case, same workspace, same day: two more tasks launched at 15:12:32 / 15:12:33 ran ~10 and ~11 minutes — longer than the one that died — and both ended with task_complete. No session in that workspace ended during that window. So this is not a duration or size limit.

    The exit code was 0 and the job record was untouched, exactly as described in the issue body: cleanupSessionJobs filters by sessionId, but sendBrokerShutdown / teardownBrokerSession do not. The broker is keyed by workspace alone (resolveBrokerStateFile → resolveStateDir(cwd)), so it is shared while its teardown is not guarded.

    For anyone hitting this before a fix lands: the loss is total but avoidable — a task told to write its output incrementally keeps whatever it had written, while one that buffers everything to the end loses all of it. In our case ~6 minutes of research evaporated because the write step was the last step.

    Suggested guard for defect 1

    Skipping teardown while another session still has a live job. Ordering matters: cleanupSessionJobs moves above the shutdown so the session's own jobs are reaped first, and only then is the shared broker considered.

    function isProcessAlive(pid) {
      if (!Number.isInteger(pid) || pid <= 0) return false;
      try { process.kill(pid, 0); return true; } catch { return false; }
    }
    
    function hasLiveJobsFromOtherSessions(cwd, sessionId) {
      if (!cwd) return false;
      const workspaceRoot = resolveWorkspaceRoot(cwd);
      const stateFile = resolveStateFile(workspaceRoot);
      if (!fs.existsSync(stateFile)) return false;
    
      return loadState(workspaceRoot).jobs.some((job) => {
        if (job.sessionId === sessionId) return false;
        if (job.status !== "queued" && job.status !== "running") return false;
        // A dead pid means the job is stale — don't let it pin the broker forever.
        return job.pid == null || isProcessAlive(job.pid);
      });
    }

    and in handleSessionEnd:

      const sessionId = input.session_id || process.env[SESSION_ID_ENV];
    
      cleanupSessionJobs(cwd, sessionId);              // own jobs first
      if (hasLiveJobsFromOtherSessions(cwd, sessionId)) return;   // broker is shared
    
      if (brokerEndpoint) await sendBrokerShutdown(brokerEndpoint);
      teardownBrokerSession({ … });
      clearBrokerSession(cwd);

    The pid-liveness check is what keeps this from becoming issue #543 — without it, a single stale running record (which this very bug produces) would pin the broker forever.

    Verified against a scratch git workspace with a synthetic state.json / broker.json:

    Other session's job Expected Result
    running, live pid broker survives ✅
    completed teardown ✅
    own session only, running teardown ✅
    running, dead pid (stale) teardown ✅

    This only addresses defect 1. Defects 2 and 3 are the reason it was silent rather than merely fatal, and they remain: a transport drop mid-turn still can't settle state.completion, and nothing pid-checks a job afterwards. A guard here narrows one common trigger; racing state.completion against client exit is what would make the whole class loud instead of invisible.

  3. astradevkin commented on Aug 10, 2026

    @astradevkin

    Confirming independent reproduction of this exact mechanism (plugin v1.0.6, macOS) — landed on the same root cause chain before finding this issue: session-lifecycle-hook.mjs::handleSessionEnd calls loadBrokerSession(cwd) unconditionally and tears down the broker with no check for other sessions' in-flight jobs; rejectCompletion is constructed in lib/codex.mjs (~L304-321) but never called anywhere in the codebase, so the dying connection resolves nothing, the event loop drains, and the process exits 0 with the job frozen at status: "running".

    One data point that might sharpen the writeup: the killer session does not need to have used Codex at all. In our reproduction, the session that ended (via /clear) had zero entries in its own job state — it had never called the companion. Its mere presence in the same workspace cwd was sufficient to trigger handleSessionEnd → broker teardown. Worth calling out explicitly since "avoid running Codex in multiple panes" is not a workaround — any sibling Claude Code session in the same directory is a hazard, whether or not it touches this plugin.

    Also worth noting for whoever picks up the fix: cleanupSessionJobs right next to the teardown call (same file) correctly filters by sessionId before acting — the asymmetry is that the broker teardown itself has no equivalent filter. The fix pattern already exists in the same file, just not applied to the broker-shutdown path.

  4. Shepherd-Creative commented on Sep 28, 2026

    @Shepherd-Creative

    Another trigger for this chain, and it needs no second session: a claude -p launched from inside the Codex job itself. With it, every job longer than ~20 s died on a single-session machine.

    Setup: plugin 1.0.6, codex-cli 0.156.1 and 0.157.0, Claude Code 2.1.281, macOS arm64. ~/.codex/config.toml includes a local peer-discovery MCP server (claude-peers) that runs a headless summariser when it starts, with no cwd, so it inherits the Codex task's:

    Bun.spawn(["claude", "-p", prompt, "--model", "haiku", "--no-session-persistence"], { env })

    What happens: Codex starts that server for the task thread. The headless claude -p is a full Claude Code session in the task's workspace, so it loads the user's Claude plugins, this one included. When it exits 17 to 22 s later, its SessionEnd runs session-lifecycle-hook.mjs, which resolves the same state dir (same git toplevel, same Claude-side CLAUDE_PLUGIN_DATA), finds the running job's broker in broker.json and shuts it down. From there it is defect 2 of the issue body: turn_aborted (reason interrupted) in the rollout, the companion exits 0 and the job stays running.

    Evidence: a preload in the broker snapshotted the process table the instant broker/shutdown arrived, which caught the sender still connected:

    node ~/.claude/plugins/cache/openai-codex/codex/1.0.6/scripts/session-lifecycle-hook.mjs SessionEnd
      <- claude -p "You generate brief summaries of what a developer is working on ..."   (started 20:19:48)
      <- bun ~/Services/claude-peers-mcp/server.ts
      <- codex app-server
      <- node .../app-server-broker.mjs serve ...
      <- node .../codex-companion.mjs task --fresh ...
    

    Same run, UTC: broker/shutdown received at 20:20:08.908; the broker ends its client sockets and closes the app-server's stdin at 20:20:09.001; the companion exits 0 at 20:20:09.002; the rollout records turn_aborted at 20:20:09.086 (duration_ms: 20648); the hook then SIGTERMs the broker pid at 20:20:09.168 (the #743 path). It happened in 4 of 4 runs: two real jobs and two reproductions on a trivial read-only task, each 17 to 22 s after the MCP server started.

    Why this shape is worth having on record

    • It is not a race. Codex took 16 to 20 s to start here (MCP servers plus plugin hooks), so the kill always landed just after the first assistant message. Through codex:codex-rescue the relayed "result" was that preamble. Through adversarial-review it was the schema-shaped commentary message, {"verdict":"needs-attention","summary":"I'm tracing ...","findings":[],"next_steps":[]}, which reads as a real verdict with no findings.
    • Extending the point above that the killer never has to touch Codex: it need not be a session the user opened either. Anything that runs claude -p in the workspace (an MCP server, a hook script, a summariser) counts, including a child of the Codex job itself.
    • Codex also runs this plugin's hooks when the plugin is installed into Codex (~/.codex/plugins/cache/openai-codex/codex/1.0.6/). Those copies ran SessionStart and SessionEnd in every one of my runs but resolved a different state dir, because Codex sets CLAUDE_PLUGIN_DATA to ~/.codex/plugins/data/codex-openai-codex. Harmless here, but a fix that changes how the state dir is resolved could let them reach the same broker.

    Workaround for this trigger: stop the child session starting. For claude-peers that is CLAUDE_PEERS_V2_SUMMARY_MODE = "disabled" under [mcp_servers.<name>.env]; with it, a job forced to 67 s completed normally. The checks proposed above and in #612 would cover it too, since the headless session has its own session_id and no jobs of its own.

  5. minhoyooDEV commented on Sep 29, 2026

    @minhoyooDEV

    Adding frequency data and a PR-triage note from macOS (plugin 1.0.6, codex-cli 0.156.1, Claude Code 2.1.284, Darwin 25.6 arm64).

    How often: of 83 Codex threads started from Claude Code in September, 3 ended in turn_aborted / interrupted, all in the one workspace where several Claude sessions run side by side. 2 of the 3 line up with another session ending:

    • abort at 12:03:35.6 UTC; another session in the same workspace started via /clear at 12:03:36.4 (0.8 s later)
    • abort at 06:55:23 UTC, inside the ~27 s in which a background Claude session in the same workspace was shutting down

    Downstream cost: the waiting session kept polling status on the orphaned job (dead pid, running) for ~27 minutes, until the job was cancelled by hand.

    Which open PR covers this case: the killed jobs were foreground codex:codex-rescue tasks of a different session.

  6. tosoyn commented on Sep 29, 2026

    @tosoyn

    Another independent confirmation (plugin 1.0.6, Linux, Claude Code with four profiles and several concurrent sessions per workspace), plus a minimal isolated repro that needs no Codex at all.

    Repro, no real broker needed. In a temp git workspace with a temp CLAUDE_PLUGIN_DATA, write broker.json for that workspace pointing at a stand-in process that leads its own process group (setsid sleep 300), just like a real detached broker. Then run the hook as a different session:

    printf '{"session_id":"OTHER-SESSION","cwd":"%s","hook_event_name":"SessionEnd"}' "$WS" |
      (cd "$WS" && CLAUDE_PLUGIN_DATA="$PD" CLAUDE_PLUGIN_ROOT="$ROOT" \
         node "$ROOT/scripts/session-lifecycle-hook.mjs" SessionEnd)

    The stand-in dies (kill(-pid, SIGTERM) from terminateProcessTree) and broker.json is removed, although the ending session never used that broker. Control run without broker.json: the process survives. SessionEnd also fires on /clear, so clearing one session is enough to kill a neighbour's in-flight job.

    The other side: leaked brokers. When a session is killed without SessionEnd (terminal closed), its broker stays forever. On this machine three brokers from 2026-09-23 were still alive 5.5 days later, each a process group of 12–17 processes (broker, codex app-server, and every MCP server from ~/.codex/config.toml: qmd ×2, chrome-devtools-mcp, runpod-mcp, node_repl, a Python MCP), about 528 MB RSS together. All their jobs had finished on 09-23 and no Claude session had those directories as cwd.

    Both sides point to the same fix: the broker needs an owner count (or per-session leases) and an idle timeout, and SessionEnd should only shut it down when no other session holds a lease.

  7. rhristow commented on Oct 7, 2026

    @rhristow

    Still happens on plugin 1.0.6 with Claude Code 2.1.292 (Linux). Deterministic repro: in a git repo, start a codex-task --effort low that runs sleep 45, then run claude -p "Reply with ok." in the same directory. The task exits 0 mid-command with no answer and its job stays "running". With --settings '{"enabledPlugins":{"codex@openai-codex":false}}' on the second session, the task completes. Open fixes that would close it: #623, #491.

  8. LukeSal88 commented on Oct 8, 2026

    @LukeSal88

    Another independent reproduction with exact timing (plugin 1.0.6, Claude Code 2.1.293, macOS, 2026-10-08).

    In a scratch git worktree with no other Claude Code session in it, a task --background was
    running. A throwaway claude -p "…" then started and exited in the same directory. The task's
    rollout recorded turn_aborted (reason interrupted) at 06:18:57.198Z, the second that
    claude -p exited, and status <job> kept reporting running until we cancelled the job.

    A workaround we use until #491 or a similar fix lands, in case it helps others: after each
    status <job> --wait cycle, read the last record of
    ~/.codex/sessions/**/rollout-*-<threadId>.jsonl. If it is turn_aborted and its turn_id
    equals the job's turnId, cancel the job and resume the thread once with
    task --resume-last --background. Matching the turn id stops an earlier abort on the same
    thread from counting against the resumed job.

  9. tatsumiyama commented on Oct 15, 2026

    @tatsumiyama

    We hit the same thing on Windows 11 with plugin 1.0.1 (two sessions in one working directory). Two details that may help:

    1. After the shutdown, the broker process stayed alive with no app-server child, and its named pipe refused connections (ENOENT). That is consistent with shutdown() waiting at await server.close() (app-server-broker.mjs line 107) with no socket destroy or timer. The leftover broker.json, broker.pid and broker.log suggest the SessionEnd hook was cut off before its teardown step.
    2. By reading the code, a broker whose app-server has gone can still look healthy. AppServerClientBase.handleExit does not set this.closed (lib/app-server.mjs lines 161-174), and ensureBrokerSession only checks that the endpoint accepts a connection (lib/broker-lifecycle.mjs lines 102-111). It also never stops the broker it replaces, because the only caller passes no killProcess (lib/app-server.mjs line 323). Separately, we saw requests sit at "Starting Codex task thread." without recovering. We have not tied that to the ENOENT state above, and by the code a stall most plausibly needs a broker that still accepts connections while its app-server is gone.

    Suggestions in addition to reference counting: refuse or defer broker/shutdown while a request is active, bound shutdown with a socket destroy and a timer, set closed in handleExit, and health-check the broker with a lightweight RPC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions