Skip to content

macOS Desktop: completed recurring automations retain CUA node_repl workers #35582

Open

Description

@sookyeomKim

Summary

Completed recurring automations leave
/Applications/ChatGPT.app/Contents/Resources/cua_node/bin/node_repl
workers alive for hours. These are not zombies: they remain sleeping (S) and
are direct children of one codex app-server.

Environment and evidence

  • macOS, ChatGPT Desktop bundle
  • Parent command:
    /Applications/ChatGPT.app/Contents/Resources/codex -c features.code_mode_host=true app-server --analytics-default-enabled
  • Observed during investigation: 284 node_repl workers, all S, all direct
    children of the same app-server PID
  • Worker elapsed times ranged from about one to eight hours
  • Earlier observation: 268 workers and 806 automation session records that day
  • Worker creation times correlated with recurring-automation executions
  • The executable is a private Desktop resource:
    /Applications/ChatGPT.app/Contents/Resources/cua_node/bin/node_repl

Public Codex repository investigation

The public openai/codex checkout has no cua_node or node_repl source
reference. Its generic code-mode host runner can accept an external host path,
but it is designed to reuse one live host connection per ThreadManager and
already has child kill + wait/reap handling when that shared connection is
dropped.

That model does not account for hundreds of direct node_repl children under
one app-server. The concrete CUA executable selection and per-automation
lifecycle appear to be in the Desktop bundle/private CUA runtime.

This is adjacent to #12491, but it should not be assumed to be fixed by #19753:
#19753 explicitly addresses stdio MCP server shutdown, whereas the observed
workers are cua_node/bin/node_repl direct app-server children.

Reproduction

  1. Restart Desktop and record the baseline node_repl count.
  2. Run a short recurring automation repeatedly and wait for each run to report
    completion.
  3. After each completion, record:
    ps -axo pid,ppid,state,etime,command |
      rg '/Applications/ChatGPT\.app/Contents/Resources/cua_node/bin/node_repl'
  4. Wait at least one hour and compare with the baseline.
  5. Expected: no job-owned worker remains after its run reaches a terminal state.
    Actual: the count rises and completed-run workers remain sleeping.

Requested owner investigation

Please route to ChatGPT Desktop / CUA runtime owners, with Codex app-server
code-mode maintainers consulted for integration.

Instrument the private CUA spawn site with:

  • automation job/run ID, spawned PID, parent PID, and spawn call site;
  • terminal reason: complete, cancel, error, channel close, or app-server stop;
  • terminate result and wait/reap result;
  • a bounded per-job registry count.

Do not use global pkill. If workers are job-owned, add a regression test that
runs several short jobs and asserts each owned process exits and is reaped on
all terminal paths. If the runtime is intentionally shared, assert a bounded
single-instance count and prove it is reused rather than recreated per run.

Related

Activity

  1. github-actions commented on Jul 27, 2026

    @github-actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  2. gkze commented on Jul 27, 2026

    @gkze

    Adding a macOS datapoint that confirms the retention behavior, plus a downstream consequence I haven't seen documented in any of the related issues (#26984, #27601, #35485): the leaked pipes exhaust the global XNU pipe-buffer pool, which deadlocks unrelated processes on the machine — software that has nothing to do with Codex — at FD counts nowhere near any per-process limit.

    Environment

    ChatGPT Desktop 26.721.41059 / build 5848
    Bundled CLI: codex-cli 0.146.0-alpha.3.1
    Codex Framework 150.0.7871.128
    macOS 26.5.2 (25F84), Darwin 25.5.0, arm64
    

    Retention datapoint (matches this issue)

    • 36 node_repl hosts under one codex -c features.code_mode_host=true app-server after ~2.5 h of light use, all sleeping (S), ~0.08 s total CPU each.

    • Spawn times cluster in bursts of 2–3 within seconds (e.g. 11:20:50, 11:20:58, 11:21:05), consistent with recurring-automation runs rather than interactive threads.

    • Only 3 of 36 had live workers (node kernel.js --session-id … + codex app-server --listen stdio://) under them; the rest were childless and hours old.

    • Each retained host pins ~13 pipe ends: ~5 on the app-server side (stdio + control pipes) and ~8 in the host.

    • The node_repl hosts are only the visible tip. Enumerating all children of the app-server shows every configured stdio MCP server and every enabled plugin's local MCP shim is retained the same way — roughly 50 full generations of the entire roster (~600 direct children), one generation per thread/automation run:

       52  uv tool uvx pagerduty-mcp
       52  uv tool uvx mcp-proxy-for-aws@latest …
       52  uv tool uvx markitdown-mcp@0.0.1a4
       52  …/1Password.app/Contents/MacOS/onepassword-mcp
       51  npm exec dotenv-cli … -- uvx mcp-proxy --transport streamablehttp …
       51  node …/bunx-…-mcp-remote@latest/…
       48  node …/bunx-…-@vantasdk/vanta-mcp-server/…
       45  node ./mcp/server.mjs --stdio     (plugin shims)
       45  node ./mcp/server.mjs
       45  node ./mcp/server.cjs --stdio
       20  npm exec <private mcp server>
       20  npm exec convex@latest mcp start
       18  npm exec xcodebuildmcp@latest mcp
       13  …/ChatGPT.app/Contents/Resources/cua_node/bin/node_repl
      

      Each generation is ~20 processes (plus uvx/npx grandchildren). The app-server's own pipe FD count (1,701 at the snapshot above) matches ~3 stdio ends × ~570 retained children almost exactly — i.e. the same never-reaped per-thread lifecycle as this issue's node_repl workers, applied to every stdio MCP server. This looks like the same root cause class as MCP stdio servers leak pipe fds + orphan child processes → cumulative EMFILE ("Too many open files", os error 24) #26984 / MCP server processes leak: per-thread processes never cleaned up (9+ GB RSS) #30408, observed from the Desktop app-server.

    Previously-undocumented consequence: global pipe-pool exhaustion deadlocks third-party processes

    XNU allocates pipe buffers from a small fixed global pool. With the app-server holding ~1,600 pipe ends (plus the retained hosts' ends), the pool exhausts, and new pipes are granted almost no buffer. At that point any process on the machine that writes a modest amount into a fresh pipe before a reader exists blocks in write() forever.

    The classic victim is a bash heredoc: bash writes the document into a pipe before exec'ing the command, assuming pipe-buffer capacity. Captured from a hung, completely unrelated process (direnv/nix-direnv sourcing an .envrc):

    execute_disk_command
      do_redirections
        do_redirection_internal
          write   ← blocked indefinitely; target pipe granted ~0 buffer
    

    Reproducible probe while the app-server holds ~1,600 pipe FDs (sizes in bytes):

    heredoc  700B: HANGS  (bash small-doc path: direct write into pipe)
    heredoc 4000B: ok     (bash switches to temp-file path, no pipe involved)
    heredoc 16000B: ok
    

    Practical impact here: every direnv export on the machine wedged permanently (nix-direnv writes its ~700 B reload script via heredoc), across multiple unrelated repos and terminals. Wedged writers never recover even after the pool frees — the pipe's buffer never grows and no reader ever appears — so each occurrence also requires hunting down and killing stuck processes.

    Two properties make this failure mode nasty to attribute:

    1. It breaks processes with no relationship to Codex, with no error — just a hang.
    2. It happens at ~1,600 FDs, far below ulimit -n-style limits, so none of the usual EMFILE symptoms from MCP stdio servers leak pipe fds + orphan child processes → cumulative EMFILE ("Too many open files", os error 24) #26984/macOS: Codex Desktop/app-server appears to exhaust syspolicyd, causing spctl “Too many open files” globally #27662 appear first.

    Evidence supporting reap-on-completion as the fix

    SIGTERM-ing a childless (completed) node_repl host is handled cleanly today: the app-server notices the exit and promptly closes its side's pipe ends (parent FD count drops within seconds, verified repeatedly). So the graceful-teardown path already works — it's just never initiated. A reap when the run reaches a terminal state (or an idle timeout) would fully resolve the retention side.

    Interim workaround for anyone else hitting this — cull childless hosts older than a few minutes (completed runs; the live ones keep their kernel.js worker as a child):

    for as in $(pgrep -f 'features.code_mode_host=true app-server'); do
      for pid in $(pgrep -P "$as" -f node_repl); do
        [ "$(pgrep -P "$pid" | wc -l)" -eq 0 ] || continue   # has worker => active
        et=$(ps -p "$pid" -o etime= | tr -d ' ')
        case "$et" in
        *-* | *:*:*) ;;                                      # hours/days old
        *) [ "${et%%:*}" -ge 3 ] || continue ;;              # MM:SS, keep < 3 min
        esac
        [ "$(ps -p "$pid" -o ppid= | tr -d ' ')" = "$as" ] && kill "$pid"
      done
    done

    Note this only slows the bleed: the retained per-thread MCP server generations (the ~600-child table above) dwarf the node_repl share, and there is no safe external heuristic for distinguishing a dead thread's MCP servers from a live one's — so the pool still exhausts eventually and only an app-server restart resets it. A reap-on-thread-termination (or idle timeout) covering all per-thread stdio children, not just node_repl, appears to be the complete fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions