Repository navigation
[Windows Codex app] Local stdio MCP servers are repeatedly spawned and not reaped within a single task #38754
Description
What version of the Codex App are you using (From “About Codex” dialog)?
Codex app: 26.810.6296.0
What subscription do you have?
ChatGPT Plus
What platform is your computer?
Microsoft Windows NT 10.0.26200.0 x64
What issue are you seeing?
Within one Codex task, each new turn/tool environment starts another full
set of local tool processes without terminating previous instances.
Observed after restart:
- qq_mail_mcp: 6 logical MCP instances / 12 Python processes
- node_repl.exe: 6 instances
- all instances share the same codex.exe app-server parent
- creation timestamps increase sequentially within the same task
With cua-driver enabled, each leaked mcp --direct instance also creates a
full-screen Cua.AgentCursorOverlay. At 12-14 instances, the Windows cursor
stutters severely. Restarting Codex clears the processes and restores smooth
cursor movement. Disabling cua-driver prevents those overlay processes but
other stdio MCP instances still accumulate.
What steps can reproduce the bug?
-
On Windows, configure at least one local stdio MCP server in Codex. In my case, I had:
- qq_mail_mcp
- cua-driver using
mcp --direct
-
Fully exit and restart the Codex desktop app so that no old MCP child processes remain.
-
Open a single Codex task. Do not create additional tasks.
-
Send several consecutive messages in the same task that require tool-enabled turns, for example asking Codex to run a read-only PowerShell command.
-
After each turn, inspect the child processes of
codex.exein Task Manager or PowerShell. -
Observe that every new turn/tool environment starts another instance of each configured stdio MCP server, while the instances created by previous turns remain running.
-
After six turns in one task, I observed:
- 6 logical
qq_mail_mcpinstances (12 Python processes because each instance has a launcher and runtime child) - 6
node_repl.exeinstances - All instances had the same
codex.exe app-serverparent.
- 6 logical
-
When
cua-driver mcp --directwas enabled, every leaked Cua instance also created a visible full-screenCua.AgentCursorOverlaycovering the dual-monitor virtual desktop. -
After approximately 12-14 accumulated Cua instances, the Windows mouse cursor started stuttering severely in normal applications.
-
Fully exiting and restarting Codex removed all accumulated child processes and immediately restored smooth cursor movement. Disabling the Cua MCP prevented its overlay processes from returning, but other stdio MCP instances continued to accumulate.
What is the expected behavior?
Codex should reuse the existing stdio MCP connection within the same task, or terminate the previous MCP child process when its tool/session context is disposed.
Additional information
No response
Activity
github-actions commentedon Aug 15, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
najy97 commentedon Aug 15, 2026
macOS reproduction, root cause, and tested patch available
I reproduced the same per-turn stdio MCP accumulation on macOS in the ChatGPT desktop app (bundled Codex app-server, Codex CLI
0.148.0-alpha.9).Reproduction evidence
Using
ps -axo pid,ppid,etime,stat,command, I confirmed that the accumulating processes were direct descendants of the long-lived ChatGPT-ownedcodex app-server, rather than Hermes, a LaunchAgent, or another watcher/service.A real session accumulated approximately:
node_replprocessesAfter terminating only those verified app-server-owned children, a few additional tool-enabled turns created new stdio MCP /
node_replpairs that remained alive. A separately parented instance of the same MCP server continued running, confirming that process-name-wide cleanup would be unsafe.This appears to be the same cross-platform lifecycle problem reported here: a single desktop task repeatedly creates stdio MCP generations instead of converging to a bounded baseline.
Root cause found
I traced two cooperating lifecycle gaps:
Superseded connection generations additionally need lease-aware retirement: an in-flight tool call must retain its exact connection, while the process should be cleaned up once the last lease is released.
Reference implementation
A tested reference implementation is available for maintainers to inspect or adapt:
najy97:codex/mcp-stdio-lifecycle-cleanupmain: openai/codex compare view0a8fb85691daa48072f4The focused patch:
SIGTERM -> grace period -> SIGKILL -> exit confirmationon only the owned Unix process groupImplementation map and lifecycle invariants
stdio_server_process.rsStdioServerProcessHandleclones share one retryable cleanup state and target only the process group, Windows Job Object/process handle, or executor handle captured at spawn.Idle | Failed -> Running -> Complete | Failed; every concurrent caller joins the same running attempt.connection_manager.rsruntime.rsprocess_group.rsEPERMmember fallback stay scoped to the captured process group.connection_manager_tests.rsprocess_group_cleanup.rsmcp_refresh_cleanup.rsstdio_message_limits.rsSuggested adoption order: spawn-owned process cleanup, pending reuse, lease-aware retirement/final barrier, then platform-specific regression coverage.
2026-08-20 hard-exit and cleanup-barrier update
The reference branch now includes two additional review-sized commits:
20e4d940d2: process-owned stdio cleanup across runtime and hard parent exits0a8fb85691: explicit connection leases, retryable retirement, and the final session cleanup barrierAdditional lifecycle hardening:
Arc::strong_countpolling for in-flight calls, resources, and event streamsSIGKILLstill triggers owned process-groupSIGTERM, bounded wait, andSIGKILLCurrent validation on macOS:
SIGKILLcodex-mcp: 183/183codex-utils-pty: 28/282026-08-21 latest-main rebase and native Windows validation
The branch is now rebased onto upstream
daa48072f4withrmcp3.1.3 and the new hosted MCP event-streaming path preserved alongside explicit connection leases.Native Windows validation on exact commit
0a8fb85691:cargo build -p codex-cli --bin codex: passcodex-utils-pty: 23/23 passcodex-rmcp-client: 251/251 pass, 7 skippedcodex-mcp: 190/190 passtest_stdio_server.execount0 -> 0after every runLegacyandV20260728protocol modes covered across success, failure, timeout, and cancellationCODEX_CLI_PATHProcess/User/Machine state, and permanent User/Machine PATH hashes remained unchangedProcess enumeration was used only as an after-the-fact baseline assertion. Cleanup itself remained scoped to spawn-captured Job/process ownership.
Validation
The patch has been validated with:
node_repl, then both reached zero on app-server shutdown while the separately owned service process survivedcodex-rmcp-client231/231,codex-mcp182/182,codex-utils-pty22/22, plus the focused core integration test1e3286d51f:codex-utils-pty21/21,codex-rmcp-client226/226, andcodex-mcp183/1833b45c29062(mainat validation time, includingrmcp3.1.2); the lifecycle-scoped run passed 449/450 tests and the focused core integration test passed, with the sole unrelated OAuth redirect failure reproduced identically on the upstream basegit diff --checkThe Windows verification used spawned-process handles/Job ownership for cleanup. Exact-path process enumeration was used only as an after-the-fact assertion that the baseline returned to zero.
Related reports include #17832 and #12491.
Under the repository's current contribution policy, external code contributions and pull requests are not accepted, so I will not open one. I am leaving the branch and validation evidence available as a technical reference, and I would be happy to provide additional diagnostics or test a maintainer-provided candidate build on macOS and Windows.
Lalita061985 commentedon Aug 16, 2026
Corroborating on macOS Desktop, with a controlled reproduction and a regression window.
Environment: macOS (Apple Silicon, 36 GB). ChatGPT/Codex Desktop
26.810.41047→26.810.52044, embeddedcodex-cli 0.148.0-alpha.9; ~18 configured MCP servers (mostly stdio). Also tested: signed26.721.81911(embedded0.146.0-alpha.3.1).Generation leak (this issue): With a single visible task, every recorded
task_started/task_completeboundary spawned a complete new stdio MCP generation (~14 processes) with the prior generation retained — 9 generations in ~31 min, timestamps matching boundaries to the second. Incident peak: ~107 complete suites / ~1,500 children / ~41 GB RSS under one app-server, ending in a kernel watchdog reset. The leak also reproduces on the older0.146.0-alpha.3.1line (5 generations in ~10 min), so it predates the current release line.New controlled finding — reaping strands the task (confirms the #28704/#35486 cross-refs): We ran an external reaper to contain memory (SIGTERM stale generations, newest preserved). A same-task/new-task A/B with timestamps: after a sweep, the existing task's tool calls fail in ~6 ms with
Transport closedpermanently, while a newly created task (fresh generation) passes the same calls. This matches the source: stdio transports are excluded from retry/reconnect (codex-rs/rmcp-client/src/streamable_http_retry.rs—PendingTransport::Stdio => false), so any workaround that kills leaked children permanently kills those tasks' tools. Operators are stuck: reap and strand tasks, or don't and hit the watchdog.Regression window that may help bisecting: embedded
0.147.0-alpha.6.5ran multi-agent sessions against the same 18-server MCP config for days with no melt.0.148.0-alpha.9additionally fails all stdio tool calls even after a fresh cold restart (single generation, direct JSON-RPCinitialize/tools/listagainst the same wrappers passes) — that fresh-start transport failure does not reproduce on0.146.0-alpha.3.1.Aggravator: per #37548, the app re-enables Sparkle auto-update on every launch; the working
0.147host bundle was overwritten in place, and there is no official downgrade path — so recovery from this regression required living with the older, still-leaking line.Asks: (1) consider inviting @najy97's prepared lifecycle-cleanup branch as a PR — it targets exactly the retained-generation mechanism; (2) stdio reconnect/rebind parity with streamable HTTP, or a documented lifecycle contract; (3) a named candidate build — we have a deterministic 10-boundary qualification harness and will report results back.
VVi3ard commentedon Aug 16, 2026
Дополнение: воспроизводится на stateful Java stdio MCP и ломает состояние workspace.
Окружение:
java -Xmx8g -jar bsl-language-server-1.0.8-exec.jar -c ../.bsl-language-server.json mcpПосле чистого перезапуска Codex был один дочерний Java PID 47376. Один вызов
list_workspace_foldersсоздал второй PID 42824; оба остались живы. В предшествующем прогоне накопилось 11 Java-процессов, созданных с 22:52:51 по 22:57:36; все имели общим родителемcodex.exe.Фрагмент
C:\Users\volos\.codex\logs_2.sqlite:Для трёх последовательных инициализаций в одном Codex process:
BSL получает от Codex:
Последствие:
register_workspace_folderхранит root только в конкретном экземпляре BSL. При следующем создании процесса список workspace пуст, поэтому stateful MCP-инструменты невозможно использовать надёжно. Тот же JAR в Claude Code работает без этой проблемы.Ducheved commentedon Aug 17, 2026
I can reproduce this on a newer Windows build of the Codex App, and in my case the accumulation eventually causes severe system-wide stuttering/freezing as system commit approaches the limit.
Environment:
26.810.7004.010.0.26200.0 x64I captured a live process/memory snapshot while Windows was already starting to stutter.
The Codex process tree had accumulated exactly 17 copies of several configured local MCP/runtime processes.
Direct children of the Codex process roots included:
npx.exenode.exenode_repl.exepython.exeThe repeated groups included Playwright MCP, Chrome DevTools MCP, Context7, filesystem MCP, node_repl, and a local Python MCP server.
The repeated count of exactly 17 instances across multiple configured runtimes is particularly suspicious. This does not look like a single leaking Node heap. It looks like complete MCP/runtime generations are being created repeatedly and retained instead of being reused or reaped.
At the time of the snapshot, the whole system had:
node.exesystem-wide accounted for:I am not claiming that all 162
node.exeprocesses belonged to Codex, because other development applications were also running. The direct-child attribution above is from a PID/PPID snapshot and is the portion I can attribute directly to the Codex process roots.Kernel pool usage was also elevated during the incident:
For comparison, a previous clean post-reboot baseline on the same machine was approximately:
So combined kernel pool usage was roughly 2.3x above that previous baseline.
However, this incident looks different from a previous Codex-related failure I captured on the same machine.
In that earlier incident, the dominant symptom was kernel paged-pool growth:
Tokealone reached ~12 GiBSeAt,SeTd,SeDt, andSeTlwere also elevatedgit.exe,powershell.exe,conhost.exe,cmd.exe, andnode.exelaunches associated with CodexAfter reboot in that earlier case:
Tokereturned to approximately 7–8 MiBThe current incident is therefore not dominated by the same extreme kernel
Tokegrowth.This time, the much larger contributor is accumulated user-space process private commit: approximately 46 GiB of aggregate private bytes with system commit at approximately 54.3 / 59.9 GiB.
The machine becomes progressively less responsive as this state accumulates. By the time of the snapshot, system commit was above 90% and Windows was visibly stuttering/freezing.
This occurred during a long-running Codex workload rather than immediately after startup. The MCP/runtime process population appears to accumulate over time instead of returning to a bounded baseline when previous tool/session contexts are no longer needed.
The process snapshot strongly suggests repeated MCP/runtime generations are involved because multiple independent configured runtimes were present in exactly 17 copies at the same time.
I have preserved sanitized diagnostics from the degraded state, including:
I can provide the sanitized snapshots if they are useful for debugging.
ALSO!
As a live workaround, I terminated several old MCP generations using
taskkill /PID <pid> /T /Fwhile leaving the main Codex app-server and active Long Horizon task running.This reduced the system from:
node.exeto:
node.exeThe active Codex Long Horizon task continued running.
This suggests that old MCP generations and their descendant process trees account for a substantial portion of the accumulated memory/process pressure, and that they can be removed independently of the active app-server/task.
tapperwijn89 commentedon Aug 17, 2026
Request for maintainer triage / canonical fix path
This issue now appears to be one of the clearest central reports for the Desktop stdio MCP lifecycle failure also seen in #38825, #38765 and #38877.
The external patch already linked above is unusually actionable: it targets pending-connection reuse, shared shutdown completion, cancellation-safe cleanup and lease-aware retirement, and it has focused validation showing repeated tool calls stay bounded at one MCP server + one
node_repl.Could a Codex maintainer please confirm whether this is aligned with the intended fix direction and, if so, invite the PR or point to the internal/public PR that supersedes it?
For affected Windows users this is now a production-blocking reliability issue rather than a cosmetic leak: related reports include system commit exhaustion, severe stutter/freezes and an OS bugcheck/reboot. A named candidate build would also be useful; I can run a controlled Windows qualification against it and report process-count/commit behavior.
terrydwisely commentedon Aug 17, 2026
Additional current-version reproduction: Python STDIO transports continuously multiply
Reporter: @terrydwisely
This is a same-machine, same-app-server reproduction on Codex Desktop
26.810.7004.0. It adds exact process ancestry, timestamps, memory impact, post-restart behavior, successful live integration tests, stale transport routing, Windows launcher behavior, an absolute-path mitigation, and independent Claude Code verification.The Outlook and Google Workspace MCP servers must remain enabled. The user is living with the leak until OpenAI ships an application-level lifecycle fix.
Windows Codex MCP process leak and duplicate-spawn report
Title
[Windows][Desktop] STDIO MCP servers multiply within one session, survive app-server restart, and bare commands can launch through cmd.exeEnvironment
OpenAI.Codex_26.810.7004.0_x640.148.0-alpha.9ChatGPT.execodex.exe -c features.code_mode_host=true app-server --analytics-default-enabledinbox-watchmanandrefresh-terry-s-morning-briefingboth pausedConfigured STDIO servers
The commands originally used bare
python. They were changed to the verified Python 3.14.2 executable above. The original config was backed up before editing.Actual behavior
There are separate duplicate-spawn, cleanup, transport-routing, and Windows-launch defects.
1. Duplicate servers inside one app-server lifetime
The current desktop app-server, PID 53644, started at 1:49:59 PM. In its first 25 minutes it produced eight MCP server launch events instead of one Outlook launch and one workspace-mcp launch. Three Outlook launches occurred within four seconds: 1:59:40 PM, 1:59:41 PM, and 1:59:44 PM.
The duplicate behavior reproduced again while this report was being prepared:
This proves the application is creating a fresh server manager/process set on a later turn or steer event instead of reusing or disposing the earlier set.
Post-restart verification at 2:33 PM
The desktop MCP restart did not correct the duplicate-spawn defect:
cmd.exeor WindowsApps Python alias launchers.This is a clean post-restart reproduction: normal application startup, an MCP restart, and ordinary integration use each retained a separate server set instead of converging on one instance per configured server.
Independent verification by Claude Code at 2:39 PM
Claude Code independently inspected the same machine and confirmed:
codex.exePID 30216 started at 2:33:16 PM.cmd.exewrappers and zero WindowsApps Python alias launchers were present.Current mitigation: living with it until OpenAI fixes it
Both MCP servers must remain enabled because Outlook and Google Workspace are required integrations. The absolute-path configuration is being retained because it removes the alias chain and target console-window launcher. There is no available configuration setting that prevents Codex Desktop from continuously spawning and retaining duplicate transport pairs, so the process leak remains active pending an application fix from OpenAI.
Continued growth while filing the issue
A Codex-side process snapshot at 2:43:39 PM found the same app-server PID 30216 still running with seven Outlook server instances and six workspace-mcp entrypoints. Additional Outlook launches appeared at 2:41:40 PM, 2:43:01 PM, and 2:43:36 PM; additional workspace-mcp launches appeared at 2:41:40 PM and 2:43:03 PM. The newest Outlook process had appeared only three seconds before the snapshot and its corresponding workspace-mcp entrypoint had not yet appeared, showing that startup is neither deduplicated nor serialized as one atomic pair. The bad-launcher count remained zero.
2. Children survive an app-server restart
An earlier app-server PID 11916 started an Outlook and workspace-mcp set at 1:23:30-1:23:35 PM. The desktop restarted its app-server at 1:49:59 PM and replaced it with PID 53644, but all 12 processes in the 1:23 set remained alive with the now-nonexistent PID 11916 recorded as the parent of the two
cmd.exeroots.Relevant desktop log excerpt:
The stale 1:23 tree was later terminated explicitly: 12 processes, 146.8 MB working set. The stale 1:59 tree was also terminated after the 2:13 duplicate appeared: 8 processes, 332.3 MB working set. No server files or configuration entries were removed.
3. Tool routing remains bound to the stale copies
After the stale 1:59 pair was terminated, the new 2:13 pair remained live and used the corrected absolute commands. However, Outlook and workspace-mcp test calls in the active task both returned:
An automatic retry did not rebind to the already-running 2:13 servers. The application spawned replacement copies but kept the active tool router attached to the older transports.
4. Bare commands use an unnecessarily deep Windows launch chain
With
command = "python", the observed workspace-mcp process tree was:The Outlook server similarly used
cmd.exe -> WindowsApps Python alias -> real Python.5. Console windows can flash
The 1:23 launches used commands such as:
Each
cmd.exehad aconhost.exechild. With Windows Terminal configured as the console host, this produced a visible terminal flash.Configuration-fix verification
An isolated app-server 0.148.0-alpha.9 probe loaded the edited config, started an ephemeral thread, and queried MCP status.
ready, server version 1.26.0.ready, server version 3.4.7.workspace-mcp.exeentrypoint ran.cmd.exeprocesses were used by either server.This verifies the absolute-path config workaround. It does not fix the desktop application's duplicate-spawn or stale-routing logic.
Expected behavior
thread/start, turn start, steer, status refresh, and config reload cannot race into duplicate launches..execommands directly with redirected stdio andCREATE_NO_WINDOW; do not usecmd.exefor executable commands.User impact
Suggested attachments
Reporting route
/in the Codex composer and choose Feedback. Include this task/session with the report.https://github.com/openai/codex/issues.https://github.com/openai/codex/issues/new/chooseand attach the report plus reviewed evidence.Related active reports found before submitting this evidence: #30408, #33946, #35485, #38526, #38614, #38925, and #38981.
5 remaining items
Fix stdio MCP process lifecycle cleanup
KimDani87 commentedon Aug 21, 2026
macOS Apple Silicon corroboration on current builds:
Local mitigation only reduced blast radius: disabled a stale duplicate registration, scoped an unrelated server out of the project, moved a credential to a mode-600 file-backed source, and performed exact-PID cleanup. It does not fix lifecycle ownership in Codex.
I can provide a sanitized process-tree capture or test a candidate build if useful.
yotamleo commentedon Aug 21, 2026
Same defect reproduced on the CLI (not the desktop app), with a per-subagent trigger and a controlled census. Filing the detail here rather than keeping a separate issue open — full write-up was #39982.
Environment: Codex CLI 0.149.0, standalone install,
windows-x86_64, Windows 11 26200. 4 configured stdio MCP servers (1 disabled). Auth ChatGPT, modelgpt-5.6-sol.What is different from the report above: your accumulation is per turn within one task. Ours is per subagent, and the server involved starts lazily (only on first tool use), so this is not eager startup — the server is spawned fresh for a subagent that actually calls it, as a new direct child of the same live
codex.exe, instead of attaching to the instance already running under that supervisor.Controlled run (two subagents, one lazily-started stdio server):
It is not one misbehaving server. After that server was removed from the config entirely, the same live supervisor still held duplicate fleets of four unrelated stdio servers (a Node REPL runtime,
uvx mcp-obsidian, and twobun-launched plugin-cache servers). An earlier snapshot on the same machine had one live supervisor holding 5 concurrent instances of a single server and a second holding 3 more. Any stdio server reachable from more than one subagent accumulates the same way.Relationship to the existing fix: #19753 ("Terminate stdio MCP servers on shutdown") is merged, and it does address shutdown. It does not cover this case, because the supervisor never shuts down — it stays live for the whole session while duplicates pile up underneath it. #38925 makes the same observation.
Why this cannot be cleaned up from outside the client: the supervisor is live the whole time, so nothing in the process tree distinguishes a redundant duplicate from a fleet in active use. There is no lease, no connection id, and no terminal state to read, so terminating a duplicate is a guess that can tear down a server another subagent is mid-call on. Our own cleanup tooling reclaims only fleets whose supervisor is gone; for duplicates under a live supervisor it is deliberately report-only — it counts them and refuses to act.
Ask (either one closes the class): reuse one fleet per active client/project so a second subagent attaches to the running server, or close stdin/the child when the owning client/subagent ends so the server sees EOF. Independently useful: expose a stable terminal state for a connection or lease, so external tooling can act on facts instead of heuristics.
Closely related, same root cause from different surfaces: #38693 (subagent-scoped trees, Desktop), #38925 (accumulation under a live app-server), #37870 (CLI, completed subagents), #38353 (proposes the pooling/teardown design).
yotamleo commentedon Aug 21, 2026
Source-level root cause, condensed here since our thread was folded into this one. Full write-up with the complete chain: #39982 (comment) — read against
main@51ebf5b1842d44a8e2c955e8b5cd2a589d41e71e.Short version: this isn't a missing teardown. Teardown is correct where it runs — it just never runs for a subagent that merely finishes.
Per-thread fleets. Every thread, root or subagent, owns its own
Session.services.mcp_runtime(codex-rs/core/src/session/mcp_runtime.rs) → its ownMcpConnectionManager→ its own stdio children. Subagents do not share the parent's.Only one reap path. Children are terminated only by
mcp_runtime.shutdown()inshutdown_session_runtime(codex-rs/core/src/session/handlers.rs:397, call at:417), reachable only viaOp::Shutdown. For a subagent that arrives from exactly two places:close_agent→shutdown_live_agent(codex-rs/core/src/agent/control/legacy.rs:8,:48)codex-rs/core/src/agent/control/residency.rs:117)A subagent that simply reaches
Completedhits neither. It stays resident on purpose so the parent canwait/resume/send_input, and its whole MCP fleet stays alive with it.Why the count climbs. Eviction is spawn-triggered only (
residency.rs:49) — nothing proactively reaps a completed resident. Capacity ismax_concurrent_threads_per_session - 1, default 4 → 3 (codex-rs/core/src/config/mod.rs:224,effective_agent_max_threadsat:1507). Steady state ≈ (live root threads) × (1 + up to 3) fleets. Our controlled two-subagent pilot took a process census 10 → 12 with nothing reaped on completion, and removing one server left other duplicate fleets standing — the duplication is per-thread, not per-server.Second-order.
impl Drop for McpServerConnection(codex-rs/codex-mcp/src/connection_manager.rs:148) only callscancel_token.cancel(); it never awaitsshutdown(). Any drop not preceded by an explicit shutdown is therefore not a guaranteed reap — relevant on Windows, where an orphaned child isn't cleaned up by process-group semantics.Suggested fix (smallest defensible). Split "resident so history/resume work" from "holds live OS processes": when a subagent reaches a terminal state — the predicate already exists as
is_unloadable(residency.rs:233:Completed/Errored/Interrupted, no active turn, no pending mailbox items) — callmcp_runtime.shutdown()and mark the runtime dormant, re-publishing on the next turn via the existingpublish_mcp_runtime/request_mcp_runtime_refreshpath. Resume, history and transcript reads never touch the MCP runtime, so nothing user-visible regresses, andensure_v2_agent_loadedalready rebuilds fully-evicted threads — a dormant-runtime resume is strictly smaller than what resume handles today.Workaround for anyone hitting this now: set
agents.max_concurrent_threads_per_session = 1to shrink the resident cache to zero (costs subagent concurrency), or have the parent callclose_agenton each subagent when done — that path does reap immediately.yotamleo commentedon Aug 24, 2026
Following up on my earlier root-cause comment above with a concrete patch.
docs/contributing.mdsays external PRs are not accepted, so this is a comment rather than a pull request — the branch is linked at the bottom if it is easier to read there.Read against
main@ad32eba832.Recap of the cause
Completed subagent threads are deliberately kept resident, and each resident thread owns a full MCP fleet.
shutdown_session_runtimereaps stdio children correctly, but it is only reached viaOp::Shutdown— sent fromclose_agentand from residency eviction, neither of which runs when a subagent simply finishes. So the steady state per root thread is up tomax_concurrent_threads_per_session - 1completed-but-resident subagents, each holding a duplicate fleet.The two properties conflated are "the thread stays resident so history/resume work" and "the thread holds live OS processes". The patch below separates them.
The change
Three files, ~74 lines.
1.
codex-rs/codex-mcp/src/runtime.rs— a way to retire the published generation without shutting it down. FactoredPublishedMcpRuntime::emptyout ofMcpRuntime::emptyand added:Retiring rather than calling
shutdown()matters: an out-of-bandcall_mcp_toolcan be running when a turn ends, and it holds theArc<McpConnectionSet>it captured. Retiring gives it exactly the lifetimereplace()already gives a superseded generation — which is whatrefresh_keeps_superseded_mcp_server_alive_for_in_flight_callspins down. The stdio children then exit when that last reference drops, through thekill_on_drop(true)and the Windows job object already set up instdio_server_launcher.rs, so descendants go too.2.
codex-rs/core/src/session/mcp.rs— the policy, next tomark_mcp_runtime_dirty:The guard mirrors the
is_unloadablepredicate already inresidency.rs.mark_mcp_runtime_dirty(notrequest_mcp_runtime_refresh) is deliberate: it must not schedule a prewarm, or the fleet respawns immediately. The nextrefresh_mcp_if_dirty— which every tool-call path and turn start already goes through — republishes.3.
codex-rs/core/src/tasks/mod.rs— one call site, at the end ofon_task_finished:Ordering is load-bearing: if queued work started a new turn,
active_turnis set again and the release is skipped.Root threads are untouched,
McpStartupPolicy::LazyWhenCachedstill applies to subagents, and disabled servers are unaffected because nothing about the projection changes — the next publish is the same publish that would have happened anyway.Tests
Four cases added to
codex-rs/core/tests/suite/mcp_refresh_cleanup.rs, in the style of the existing test there (real stdio server,MCP_TEST_PID_FILE, process-liveness assertions):call_mcp_toolstill holds its connection when the turn completes, and the process exits only after that call is released;What I ran
cargo clippy -p codex-core -p codex-mcp --tests -- -D warningsis clean. The suite is#[cfg(unix)], so the lifecycle tests ran on Linux even though the original report was Windows 11 / CLI 0.149.0:The last one is your existing test, included as a control that the retire path does not change superseded-generation lifetime.
Branch, if a diff is easier to read than a comment: https://github.com/yotamleo/codex/tree/fix/mcp-runtime-release-on-terminal-subagent — no PR opened, per the contributing policy. Happy to re-measure the process census against a build if that would help.
One thing I did not change
impl Drop for McpServerConnectionstill only callscancel_token.cancel()and does not awaitshutdown(). That is fine for this path because the transport iskill_on_drop, but it does mean any other drop-without-shutdown path depends on that rather than on an explicit reap.grtninja commentedon Aug 24, 2026
Windows corroboration: process/thread/handle exhaustion reaches Win32 1816
I have a separate Windows 11 long-running Codex reproduction that appears to extend this same lifecycle family from duplicate stdio/MCP children into eventual process-creation failure.
Sanitized live snapshot before cleanup
node,node_repl, Python bridge/MCP helpers, and Codex-owned/helper descendantsNot enough quota is available to process this commandThose aggregate counts are system-wide; I am not claiming every process/handle is Codex-owned. The current cleanup pass is deliberately tracing PID, PPID, executable, creation time, command fingerprint, and owning task/lease before terminating anything.
The user-visible pattern is consistent with this issue: long-running Codex work creates helper generations faster than terminal task/turn cleanup returns them to baseline. Restart/cleanup temporarily restores process creation.
Current-main code audit
I also checked current public
openai/codexsource rather than relying only on the observed process tree.codex-rs/rmcp-client/src/stdio_server_launcher.rsnow has useful Windows Job Object/process-handle containment, butStdioServerProcessHandlestill uses a singleterminated: AtomicBool: the first local caller marks termination complete, signals the Job/process, and returns without that method waiting for a verified tree exit/reap; later callers return immediately once the flag is set. That still leaves a cancellation/concurrent-shutdown correctness gap of the kind described in the reference implementation already posted above.Separately,
codex-rs/codex-mcp/src/runtime.rs::replace_fresh()publishes withprevious: None, and the current session MCP refresh path callsreplace_fresh(). That means hard refreshes intentionally forgo connection reuse, making correct retirement/reaping of old generations especially important.Additional acceptance criterion suggested by the Windows failure
Please consider a Windows regression that repeatedly performs tool-enabled turns/subagents and asserts all of the following return to a bounded baseline after their owners reach terminal state:
node_repl/MCP generationsCreateProcesswithout Win32 1816Cleanup should remain spawn/lease-owned; process-name-wide cleanup would be unsafe because unrelated same-name Node/Python processes can be active at the same time.
I am preserving the detailed local process census privately and can add further sanitized before/after measurements once the ownership-bound cleanup run completes.
grtninja commentedon Aug 24, 2026
Follow-up after current-main history review: this reproduces after the existing Windows process-tree fixes
I checked the relevant landed history so this report does not ask for work Codex already has:
9daa491f7c27a5513fec554473a7122d88fca367/ Harden local MCP server process tree cleanup #37366 (2026-08-07): Harden local MCP server process tree cleanup — Windows non-breakaway Job Objects, process-handle fallback, descendant cleanup tests.8751fd3fcb8031d42b62670a6131872074635c9b/ Shut down superseded MCP managers on refresh #29608: explicitly shut down superseded MCP managers on refresh.4e0cf945b7f43f1f9c1d09faadac6def76bdefbf/ Terminate stdio MCP servers on shutdown to avoid process leaks #19753: explicit stdio MCP shutdown/draining on session shutdown/refresh.82c981cafc57dfed8383e72c6bbf6082622a3b4a/ Process-group cleanup for stdio MCP servers to prevent orphan process storms #10710: original process-group cleanup.So the Aug 24 Windows exhaustion reproduction should be treated as a remaining lifecycle-generation/retirement/postcondition defect, not simply “Codex lacks process-tree cleanup.”
The remaining current-main concerns are:
StdioServerProcessHandleInnerstill collapses shutdown state into oneterminated: AtomicBool; the first caller signals termination and returns, while later callers can treatterminated=trueas terminal without sharing/waiting for a verified cleanup result.McpRuntime::replace_fresh()withprevious: Noneon the current session refresh path, so any missing retirement edge has multiplicative impact.The useful regression therefore is not merely “grandchild dies when handle drops.” It is a long-lived app-server task/subagent sequence that measures active MCP generations + process/handle/listener counts and requires them to return to a bounded baseline after each owner reaches terminal state, then verifies a fresh CreateProcess still succeeds.
I’ll add exact before/after ownership-bound Windows cleanup measurements once the local census finishes.
grtninja commentedon Aug 24, 2026
Additional finding: Win32 1816 should be treated as an operation failure requiring attribution, not a machine-capacity diagnosis
A recent Windows reproduction initially surfaced Win32 1816 during a Codex-managed process launch. Further investigation showed that the machine remained capable of launching substantial workloads; the failure was specific to the Codex execution path and required tracing the actual failing operation.
Recommended reliability behavior:
A concrete local recovery path required fixing the failing component rather than relying on small targeted cleanup events. Cleanup should be considered successful only when the original capability is restored or ownership state is proven healthy.
Suggested acceptance criterion:
process launch failure -> ownership attribution -> targeted repair -> successful retryrather than:
process launch failure -> Win32 error -> assume environment blocker.grtninja commentedon Sep 1, 2026
Correction and ownership note
I found a concrete defect in my own custom Codex compatibility framework: an older line hard-coded a Windows Job Object active-process limit of 96 around the Codex app-server descendant tree.
The assumption predated the current sub agent spawning behavior and is now a known internal conflict.
Removing only that active-process limit, while preserving kill-on-close lifecycle ownership, immediately restored helper and subagent spawning without changing the workload or materially changing host load.
This is not a claim that stock Codex has a 96-process cap. It is a warning for users who run custom launchers, shims, supervisors, orchestration wrappers, or compiled compatibility binaries: check your own frameworks for stale assumptions and hard-coded limits, especially:
JOB_OBJECT_LIMIT_ACTIVE_PROCESSActiveProcessLimitOne important diagnostic caution: Codex may mistake an inherited limit from an external compatibility framework for a hard-coded OpenAI/Codex configuration because the failure surfaces inside Codex Unified Exec. That attribution can be wrong. In this case, Codex initially reasoned about an internal/OpenAI process ceiling, but the effective 96-process cap was owned by my local compatibility shim.
Before assigning ownership, inspect the live process ancestry, Job Object owner and effective limits, the exact launcher binary actually running, and any framework wrapped around the app-server. Also inspect compiled binaries when source and runtime may have drifted.
The local limit explains this machine's immediate Win32 1816 admission failure. It does not invalidate the separate upstream lifecycle questions around per-subagent MCP/runtime duplication, completed-agent residency, or helper cleanup.
MatthiasCantini commentedon Sep 2, 2026
Confirming this still occurs on Windows with Codex app
26.825.6671.0x64.After extended normal use, I observed 210 retained Node-related processes:
node_repl.exenode.exeC:\Program Files\nodejs\node.exeThe processes appeared in timestamp-aligned groups accumulated across tool-enabled turns. Together they consumed approximately 8.1 GB working-set memory while showing effectively 0% CPU activity during a three-second sample.
This caused noticeable system sluggishness. No custom Codex launcher, compatibility framework, or process-limiting wrapper is involved.
I have not yet completed a controlled restart-and-reproduction cycle, but this confirms the accumulation persists in app version
26.825.6671.0.yotamleo commentedon Sep 2, 2026
Adding a measurement of the accumulation half of this thread, since @MatthiasCantini's report above is the same shape we hit and the distinguishing detail cost us a while to find.
Setting: Windows, Codex app-server driven by MCP clients. We do run our own wrapper scripts around the client, but they impose no process cap and no Job Object — I checked ours against @grtninja's list above after that comment, and found none. The mechanism below doesn't appear to depend on the launcher.
What we measured (2026-07-27). One
codex.exe app-server, pid 51100, alive 13:39→21:12, had accumulated 45 launcher+server pairs for a single MCP plugin under it. Roughly 9 GB of resident memory accrued in well under a day. Like the reports above, the duplicates sat at effectively zero CPU.Why an orphan sweep doesn't find it. This is the part worth stating plainly: the leak is inside a still-live app-server, not behind a dead one. A long-lived app-server spawns a fresh MCP fleet per client connection and never reaps the previous one, so each duplicate is individually indistinguishable from a fleet that is legitimately in use, while collectively they are the leak. Every conventional orphan heuristic therefore correctly declines to touch them:
cxc-<token>-codex-app-serverpipe" — false here, one does;codex.exedescendant" — we tried both and they are false signals in this tree; we dropped them.We had two sweeps built on exactly those tests, and both correctly judged the tree live and skipped it every single day while it grew.
The shape that does work, and specifically not a cap. We settled on per-key supersession rather than any process ceiling. Group the direct children of a live app-server by the MCP server they actually are (the launcher's
--cwd <plugin>path where present, else the command line minus the leading exe token); within one(app-server pid, fleet key)group the newest roots are current and strictly older ones are superseded. Supersession alone is not sufficient to end a process, so three gates narrow it, and each can only ever remove candidates:We deliberately rejected bare per-plugin cardinality — "no more than N fleets per plugin" — because it kills live tooling, and it is precisely why the two conservative sweeps existed in the first place. That's the same conclusion @grtninja reached from the other direction: a fixed count is the wrong instrument for this, and removing the cap while keeping kill-on-close lifecycle ownership was the fix. Ending processes by liveness and supersession is safe in a way ending them by count is not.
This runs for us as a scheduled task rather than a daemon, and one scheduling detail generalises: it must run in the interactive session as the same user, not as SYSTEM or "run whether the user is logged on or not". A detached or elevated context cannot see candidate processes' command lines, and the fleet key is derived from the command line — so in that context the sweep silently matches nothing and no-ops every day while looking healthy. Also worth noting: a daily fire was not enough on its own, since ~9 GB accrued between two of them; we repeat every 4 hours.
Our implementation is in a private repo so there's nothing useful to link, but the test above is the whole of it and is straightforward to reproduce.
None of this bears on the Unified Exec admission failure discussed earlier in the thread — it is only about the retained-process accumulation, which the reports above suggest persists without any custom launcher involved.
CoolCollin commentedon Sep 11, 2026
Same failure mode here on Windows 11 (10.0.26200 x64), Codex CLI
0.153.4. I filed #44683 before finding this thread — closing that as a duplicate. Two pieces of evidence from my capture that I don't think are in this issue yet, since they narrow down where the bug is.1. All configured MCP servers accumulate in perfect lockstep — not a per-server crash loop
Reports so far describe a single server leaking (
node_repl,qq_mail_mcp). I have 4 servers configured across 3 different executables/runtimes, and they accumulate identically:Identical counts, identical timestamps, four unrelated binaries. A crash-and-restart loop in any individual server cannot produce this. The whole MCP group is being re-initialised as a unit and the previous group is abandoned.
This lines up with the mechanism named in #18881 ("MCP child processes leak when McpConnectionManager is replaced", closed) — it looks like the same class of bug resurfacing, or never fully fixed for this path.
2. Session rollout files line up with spawn times to the second — this identifies the trigger
Comparing
sessions/<date>/rollout-*.jsonlfile creation times against MCP process start times:So establishing a session context spawns a group.
Important caveat: that day had only 6 rollout files but 35 accumulated groups. So session creation is not the only trigger — something else (session resume, config reload, thread switch) also re-spawns the group. Spawn intervals were irregular, 5–40 min apart, with occasional pairs 0.1–1.0 min apart. That irregular pattern matches the "every turn" behaviour described in this issue rather than a timer.
3. Scale, and why it is worse than the process count suggests
4.7 hours on an otherwise idle machine (main process measured at ~0% CPU, no task running, no user interaction — growth continues regardless):
Second-order cost that isn't obvious: several Windows components periodically enumerate the whole process table (WMI
Win32_Processsubscribers, security software, search indexers). Their CPU cost scales linearly with process count, so the leak inflates system CPU well beyond what the idle leaked processes themselves consume. In my case an unrelated app holding aWITHIN 1WMI subscription onWin32_Processwas costing ~12% of one core purely because the table had doubled in size.4. One detail that affects the fix: child process trees
A
uvx-launched MCP server is not one process, it is a 5-process chain:35 groups of that one server = 175 processes, about a third of the total leak.
Killing only the direct child of
codex.exeleaves theuv/pythongrandchildren orphaned and still running. Any reaping logic has to terminate the whole process tree as a group (job object on Windows), not just the immediate child. I hit exactly this when cleaning up manually — the first pass left ~30 orphans behind.Also worth noting: that particular server connects to an external application that was not running at all during the entire session. All 175 of its processes were spinning against an unreachable target, so the leak is not gated on the server actually being usable.
Workaround in the meantime
For anyone else hitting this: killing all but the newest group does not disturb the running Codex session — I've done it twice now with no ill effect. Group the children of the
codex.exePID by command line, keep the most recent of each, kill the rest along with their descendants.Keep only the newest copy of each server in the latest conversation (#3)
soeprbp commentedon Oct 9, 2026
Additional Windows evidence after updating to
OpenAI.Codex_26.1007.2314.0_x64, with bundledcodex-cli 0.162.0-alpha.17.2.MCP process retention is still observable in this version. The inactive-window resume log flood reported in #51888 was absent from the retained main-process log for this updated app instance, while multiple MCP process sets remained resident and new sets continued to start.
Retained sets under one app-server
One continuously running
codex.exe app-serverowned 10 instances of a custom .NET STDIO MCP server created in a burst between 09:42 and 09:45. All 10 were still alive at approximately 10:27. Each had accumulated only 2.59–4.16 seconds of lifetime CPU and held 35.6–39.2 MiB working set. Matching Node/Node REPL process groups were also present.This establishes long-lived, mostly idle residency. Exact ownership of every retained server could not be mapped to a currently active or completed chat, so lifetime CPU alone is not being used as proof that each individual process was abandoned.
Further instances of that same server started at:
These are observed creation times under the same app-server, with no parent-process restart between them. The last interval was 4 minutes 23 seconds. This sample does not establish a constant 20–40-second respawn rate.
Measured memory impact
Machine-wide snapshots for the relevant executable names:
At 10:25:35, their summed working sets were approximately 7.13 GiB: 6,465 MiB Node, 273 MiB Node REPL, and 566 MiB custom MCP. These are summed working sets, not unique/private memory. The table includes all processes with those executable names; it does not claim every Node process on the machine belonged to Codex. The accumulating MCP groups were separately checked through their app-server ancestry.
The temporary plateau was followed by another set and a measurable RAM drop. Some temporary generations did exit, so the evidence does not support a claim that no MCP process is ever reaped.
Background-thread association
The 10:25:10 generation was followed by
mcp_server_startup_status_updatedevents for multiple unrelated servers sharing one new thread ID. The app's thread inspection returnedephemeral threads do not support thread/turns/listfor that ID.Other recent launches aligned with normal chat start/resume requests. This environment has multiple chats, so this report extends the retained-process/background-session evidence in this issue; it is not a clean reproduction of the original single-task scenario. The feature creating the ephemeral thread and its intended teardown policy remain unconfirmed.
Please investigate MCP manager disposal, process-tree cleanup, and idle-session retention separately from the fixed inactive-window resume loop. Exposing the owning thread and retention reason for each MCP process would make it much easier to distinguish needed servers from abandoned generations.
Conversation identifiers/content, private server names, user paths, credentials, raw logs, and local containment details are omitted.
CoolCollin commentedon Oct 12, 2026
Following up on my earlier comment in this thread. This one is about the shape of the problem across the tracker rather than new symptoms, because I think that is now the blocker.
23 linked issues, six months, zero assignees, zero maintainer replies
I pulled the complete set of issues verifiably linked to this one — cross-references on this issue, the duplicate-detector's own list, and issues cited in the comments above — and checked each through the API on 2026-10-12.
McpConnectionManageris replacednode_replworkers remain after Code Mode, correlate with system-wide UI stalls0xc0000409crashesnode.exeto ~55 GB virtual memory, crashes/reboots WindowsCLOSE_WAITsockets accumulatethread/unsubscribeWhat the table shows:
MEMBER,OWNERorCOLLABORATOR: zero, on all 23. (Private org membership would not appear in that field, so I can only speak to what the API exposes — but no comment on any of these issues is presented as coming from the team either.) On this issue the five labels were applied bygithub-actions[bot]; every non-bot participant is an external reporter.The reported consequences, taken verbatim from those titles: OS freezes (#38765), Windows crashes and reboots (#38877), a hard-frozen 16 GB Mac (#44996), an unresponsive PC (#29079), 490 processes / 26.5 GB (#38537), ~55 GB virtual memory (#38877), app-server stalls (#42740). That is lost work and forced reboots, not sluggish UI.
Two patches have been offered, and neither can be submitted
main @ 51ebf5b1842d44a8e2c955e8b5cd2a589d41e71e, with a concrete patch. The author explains why it is a comment and not a pull request:docs/contributing.mdstates "We do not accept external code contributions or pull requests."I am not arguing with that policy. I am pointing out its consequence here: the channel that could have closed this in August is closed by design, and the channel that remains has produced 23 issues and 120 comments without a reply. #38754#issuecomment-5315604818 — an explicit "Request for maintainer triage / canonical fix path" from 2026-08-17 — has gone 56 days unanswered.
Current measurements, 2026-10-12
Still reproducing on
OpenAI.Codex 26.1007.21434, Windows 11 10.0.26300 x64.Re-spawn cadence follows turns, not a timer. One session, one
codex.exe app-server, single 2.5-hour working window. Gaps between consecutive full-group re-spawns:Each group is three
cmd.exebundled-plugin launchers (launch_code_review_mcp.cmd,launch_codex_app_tools_mcp.cmd,launch_codex_security_mcp.cmd) plus their node/python children. Those 2.5 hours reached 50 retained processes / 1.02 GB under one app-server.The fix shape that works. I now run an out-of-process reaper on a 30-minute timer. Two details were load-bearing, each found only after a failed attempt:
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSEat spawn time. Terminating only the direct child ofcodex.exeorphans theuv/python/nodegrandchildren — my first pass left roughly 30 orphans behind. Closing the job handle reaps the whole tree, including grandchildren left by a root that exited normally.Two runs reclaimed 55 processes and took the tree from 50 proc / 1.02 GB down to 3 proc / 0.04 GB, with the live session unaffected. I offer this as evidence about where lifetime ownership needs to sit, not as code to adopt.
The ask
Any one of these, whichever is cheapest for you:
A single sentence on any of the three would stop the duplicate reporting. At the moment there is no signal that this is seen at all, so every few days someone else spends an evening re-measuring it and files it again.