Repository navigation
Orphaned process pairs burn CPU when terminal is closed (SIGHUP not handled) #14962
Description
Summary
When a terminal tab or window is closed while Codex is running, the Node wrapper (codex.js) and its Rust child process both survive as orphans, with the Rust process entering a tight read(fd=0) → EIO spin loop at ~103K syscalls/sec, burning ~15-19% CPU each. Each terminal close creates one orphan pair (Node + Rust), accumulating indefinitely until manually killed.
Root Cause
Three-layer failure chain:
Layer 1 — Signal suppression: codex.js registers a handler for SIGHUP (line 204) that calls forwardSignal("SIGHUP") but never calls process.exit(). Since Node's default SIGHUP behavior (terminate) is suppressed by the handler, the Node process survives. Because the session leader (Node) survives, the kernel never delivers SIGHUP to the Rust child process — SIGHUP only propagates to the process group when the session leader actually dies.
Layer 2 — PTY I/O spin: When the terminal closes, the PTY file descriptors are revoked. The Rust child's crossterm event reader calls read(fd=0) in a tight loop, each returning EIO immediately (~103K calls/sec). This is a known crossterm bug: crossterm-rs/crossterm#793 (open since June 2024). Codex uses a forked crossterm 0.28.1 from nornagon/crossterm.
Layer 3 — No application-level exit: Even if crossterm's event stream ended gracefully, the Rust process's tokio::select! loop has other active branches that keep it alive.
Reproduction
Minimal reproduction using a Python PTY script:
#!/usr/bin/env python3
"""Reproduce codex orphan bug: spawns codex in a PTY, then closes the PTY."""
import pty, os, time, signal
pid, fd = pty.openpty()
child = os.fork()
if child == 0:
os.setsid()
os.dup2(fd, 0); os.dup2(fd, 1); os.dup2(fd, 2)
os.execvp("codex", ["codex"])
else:
time.sleep(3) # let codex start
os.close(fd) # revoke PTY (simulates closing terminal)
time.sleep(2)
# Check: both Node and Rust processes should be dead, but they're not
os.system(f"ps -o pid,ppid,%cpu,command -p {child}")
os.system("ps aux | grep 'codex' | grep -v grep")Steps:
- Run the script above
- After 5 seconds, check for orphan processes:
ps aux | grep codex | grep -v grep - Observe: both the Node wrapper and Rust child survive with elevated CPU
Evidence
From a system with 6 orphan pairs (12 processes):
# lsof showing revoked PTY file descriptors on orphaned Rust process:
codex PID user 0u CHR ?,? 0t0 ??? /dev/ttys??? (revoked)
codex PID user 1u CHR ?,? 0t0 ??? /dev/ttys??? (revoked)
# dtruss showing EIO spin rate:
read(0x0, ...) = -1 Err#5 [~103,000 calls/sec]
# CPU usage per orphan pair: ~15-19% each process (30-38% per pair)
# 6 pairs = 104.7% total CPU consumed by orphans
Suggested Fixes
Priority 1: Fix codex.js SIGHUP handler (Node wrapper)
Replace the current signal registration (around line 204):
// Current (broken):
["SIGINT", "SIGTERM", "SIGHUP"].forEach((sig) => {
process.on(sig, () => forwardSignal(sig));
});
// Fixed:
["SIGINT", "SIGTERM"].forEach((sig) => {
process.on(sig, () => forwardSignal(sig));
});
process.on("SIGHUP", () => {
forwardSignal("SIGHUP");
setTimeout(() => {
try { child.kill("SIGKILL"); } catch {}
process.exit(1);
}, 3000).unref();
});The .unref() ensures that if the child exits normally before the timeout, Node proceeds through the existing exit path (lines 213-229) without the timer keeping the event loop alive.
Priority 2: Fix crossterm EIO spin (Rust/crossterm layer)
The forked crossterm (from nornagon/crossterm) should treat EIO on read() as a stream-termination event rather than retrying indefinitely. See crossterm-rs/crossterm#793.
Priority 3: Rust application-level cleanup
The Rust binary should handle SIGHUP explicitly (e.g., via tokio::signal::unix::signal(SignalKind::hangup())) and initiate graceful shutdown.
Priority 4: Defense-in-depth — detect revoked FDs
The Rust binary could periodically check if its stdio FDs are still valid (e.g., fcntl(0, F_GETFD)) and exit if they've been revoked.
Environment
- OS: macOS 13.7.8 Ventura (x86_64, Intel)
- Codex version: codex-cli 0.115.0
- Node: v25.6.0
- Shell: zsh
Activity
github-actions commentedon Mar 17, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
roazanas commentedon Jul 30, 2026
Adding some verification on top of this, because the situation upstream has changed since March and one of the three layers here no longer applies.
1. The upstream fix now exists, but is incomplete — and is not in the pinned fork
crossterm#1067 (open since 10 Jul 2026) fixes the inner Unix read loops described in Layer 2:
Ok(0)returnsUnexpectedEof, and theErrarm gainsreturn Err(e)soEIOpropagates. It also fixes two sibling loops and adds atry_readregression test.I checked the three revisions that matter, and the read loop is byte-identical in all of them — the fix is in none:
Ok(0)WouldBlock/Interruptederrornornagon/crossterm@87db8bfrust-v0.145.0,rust-v0.146.0openai-oss-forks/crossterm@f69a4a0mainSo
main's migration from thenornagonfork toopenai-oss-forks/crosstermdid not pick this up. The pin being a fork under OpenAI's control is the upside here — it can be patched without waiting on upstream review.But cherry-picking #1067 as it stands would not be enough. I built a minimal tokio +
EventStreamconsumer against the PR head and tore its pty down. The CPU spin goes away (~101% → ~0%), but the error never reaches the consumer: the stream stays pending, and the process sits idle instead of exiting. In a 5 s window after teardown, an instrumentedpoll_nextwas called 14 times and returnedOk(false)every time, neverErr. Details and the mechanism are in my comment on the PR. Applied alone to Codex, it would therefore likely replace a hot orphan with an idle one rather than triggering the shutdown path. I did not build Codex itself with the patch, so that last part is a prediction from the minimal reproduction, not a measurement.2. Layer 3 looks already solved
The issue notes that even a graceful stream end might not exit. On
rust-v0.146.0that path is handled:TuiEventStream::poll_crossterm_eventmapsSome(Err(_))andNonetoPoll::Ready(None)(event_stream.rs#L214-L219), and the main loop turns that intohandle_exit_mode(ExitMode::ShutdownFirst)with aterminal input stream closedwarning (app.rs#L1212-L1221). The Codex side is ready to consume an input error — crossterm has to hand it over first.3. macOS data point, standalone binary (no Node wrapper)
Worth adding because Layer 1 here is about
codex.js, which was not in play for my instance — the parent was already gone (PPID 1) and there was no Node wrapper anywhere in the tree. Codex0.145.0, aarch64-apple-darwin standalone release:sampleputs 2305 of 2310 samples in the same crossterm function and offset:I found two hot orphans consuming roughly one core each at the time of measurement; one had been alive for 2d11h and had accumulated 953 CPU minutes, 333 of them system time. RSS stayed around 163 MB across the observations I took, with no concurrent memory growth — the immediate failure mode was CPU saturation. A surviving process also keeps its normal runtime resources open; in the unpatched instance I saw handles to
state_5.sqlite,logs_2.sqlite,goals_1.sqlite,memories_1.sqliteand their WAL/SHM files.On the EOF-vs-EIO question: on macOS
man 2 revokedocuments that aread()from a revoked terminal device returns zero, as an explicit exception to "subsequent operations fail", whileman 2 readdocumentsEIOseparately, including for an orphaned process group. I did not trace which one my instance hit, so I can only say both are reachable and #1067 handles both.4. Reproduction
Two notes, since #16862 stalled on the maintainer not being able to reproduce by closing Terminal.app.
The Python script in this issue does not tear the pty down, so it will produce a false negative.
pty.openpty()returns(master_fd, slave_fd), so inpid, fd = pty.openpty()the variablefdis the slave; the parent'sos.close(fd)closes that slave copy and leaves the master open, so no hangup occurs. Separately,dup2-ing a slave onto 0/1/2 aftersetsid()does not make it the controlling terminal on macOS/BSD withoutTIOCSCTTY.What did reproduce deterministically for me, on stock 0.146.0:
pty.fork()(which isforkpty— it doessetsidpluslogin_tty, so the slave really becomes the controlling terminal), then in the childsignal.signal(SIGHUP, SIG_IGN)beforeexecvso the process survives the hangup, then in the parent close the last master fd. CPU measured before and after, so startup cost cannot be mistaken for the loop:Same hot stack as the orphan above. Notably the descriptors were not revoked in this harness —
fd 0/1still pointed at the live/dev/ttysNNN— sorevoke()is not a precondition; a hung-up pty with a surviving process is enough.The loop is only reachable if the process survives the teardown, which is probably why manual repro has been unreliable. On my machine, closing a kitty window with a running TUI left no orphan at all in 2 of 2 attempts — the process was simply gone. I did not verify the mechanism, so I can only report the clean exit, not its cause. Warp reproduced it twice.
5. Related reports
Same failure family. #11077, #28315 and #19958 contain direct EOF / deleted-or-revoked-PTY / hot-read evidence; the rest match on lifecycle and CPU symptoms: #16862, #13928, #28794, and closed #21021 — closed by its author on 30 May without a fix being confirmed, so it should not be read as resolved.
One small correction for anyone comparing rates: the
/proccounters in #28315 give243963619706 − 243943917947 = 19,701,759reads in ~2 s, not the1,172,137stated there — about 9.8M/s, consistent with a syscall-rate loop. (syscris process-wide, so it is strong circumstantial evidence rather than a trace ofread(0)specifically.)