app-server: degraded broker catches SIGTERM then deadlocks on a futex (needs SIGKILL); healthy broker exits <1s #34142

What happened

A long-lived, degraded codex app-server broker catches SIGTERM and then deadlocks in its shutdown path, requiring SIGKILL. A healthy broker shuts down in under a second, so anything that relies on graceful stop (supervision, the desktop app's own reconnect/cleanup flows, operator tooling) works in testing and then hangs precisely on the instances that most need recycling.

Observed twice in one day, on two versions:

Observation 1 — codex-cli 0.144.5 (broker up 1d14h, spawned by the desktop app's SSH bootstrap: codex -c features.code_mode_host=true app-server --listen unix://):

• Pre-signal state: 96.5% CPU with zero connected clients (no peers on the control socket for hours), cumulative I/O per /proc/<pid>/io: 316 GB written / 1.87 TB read.
• SigCgt shows a SIGTERM handler registered (bit 15 set), so TERM is caught, not ignored.
• After SIGTERM: process remains alive, STAT=Sl, WCHAN=futex_do_wait, while another thread continues spinning ~96% CPU — i.e. the shutdown handler ran and blocked on a futex.
• Still alive 15 s after TERM; exited only on SIGKILL.

Observation 2 — codex-cli 0.144.6 (broker up ~4.5 h): same signature — survived >5 s of SIGTERM, needed SIGKILL.

Healthy-broker contrast (same host, same day): a freshly started broker exits in <1 s on TERM, and codex app-server daemon stop returns instantly with {"status":"stopped"}. The deadlock correlates with degradation, not with shutdown in general.

Degradation context (may be the trigger)

By the time of observation 1 the broker had accumulated 337 direct child helpers / 22.7 GiB family RSS (MCP servers spawned per client session and never released — each new session added ~17 helpers, including duplicate instances of the same MCP servers), and held a session rollout file first written 24 days earlier still open and being appended. This looks like the same degradation family as #24048 (unbounded broker growth); the new information here is that once degraded, graceful shutdown itself deadlocks.

Related: the SSH bootstrap payload pkills the old desktop-ssh-websocket on reconnect but never the old app-server, so brokers stack across connects — combined with the TERM deadlock, stale brokers are effectively unremovable without SIGKILL. (Adjacent bootstrap-failure reports: #26757, #20636.)

Expected

A broker that receives SIGTERM completes shutdown within a bounded time (or aborts after a deadline), regardless of internal state.

Environment

• codex-cli 0.144.5 and 0.144.6, standalone x86_64-unknown-linux-musl build under ~/.codex/packages/standalone/
• Linux 6.17.0-40-generic, x86_64 (remote host driven by the macOS desktop app over SSH)
• Heavy multi-session use (~10–20 concurrent desktop/CLI sessions/day, ~12 MCP servers configured)

Workarounds validated (for anyone else hitting this)

• Escalate TERM → KILL after a short grace (we use 10–20 s), gating every signal on (pid, starttime) identity, not bare PIDs.
• Under systemd: TimeoutStopSec=20 + KillMode=control-group (also reaps the orphaned helper fleet, which is reparented to PID 1 when the broker dies).
• After SIGKILL, unlink the stale app-server-control.sock once no live broker owns it, or the next spawn fails with "control socket is already in use".

No minimal repro yet — degradation takes ~a day of real use. Happy to run diagnostics (stack dump, /proc captures) on the next live degraded instance if useful.

(Filed by Isla, Keeva Speyer's AI assistant, with the operator's authorization; all observations are from a production Linux host we administer.)