[Windows] Idle local STDIO MCP servers can exhaust Tokio blocking pool and stall the app-server

What version of Codex are you using?

Reproduced from upstream commit 042534ec1ab2f79c2997e779347d5383832ecb2e (Release 0.154.0-alpha.1). The same local-stdio implementation is still present on current main at the time of filing.

Platform

Windows 11 x64.

Summary

On Windows, every quiet local STDIO MCP client can occupy two workers in Tokio's shared blocking pool indefinitely: one pending read for child stdout and one for child stderr. Codex uses Tokio's default blocking-thread limit (512), so enough retained/per-thread MCP clients can prevent unrelated spawn_blocking work from ever starting and stall the entire app-server even though async network I/O can continue.

This is a separate low-level failure mode underneath the already reported MCP process/session multiplication issues. Explicit shutdown helps only after an owner is actually disposed; it does not make hundreds of simultaneously retained local STDIO transports safe on Windows.

Confirmed mechanism

Codex currently does the following:

codex-rs/arg0/src/lib.rs::build_runtime() creates a multi-thread Tokio runtime without setting max_blocking_threads (Tokio default: 512).
codex-rs/rmcp-client/src/local_child.rs starts every local MCP child with piped stdin/stdout/stderr.
The rmcp transport continuously reads stdout.
codex-rs/rmcp-client/src/stdio_server_launcher.rs starts a second continuous BufReader::lines() drain for stderr.
Tokio 1.52.3 represents Windows ChildStdout/ChildStderr as io::blocking::Blocking<ArcFile>, so each pending read consumes a shared blocking worker.

Tokio documents the exact generic failure in tokio-rs/tokio#5777:

on Windows, reading a pipe is a blocking read and will take away a blocking thread forever

and:

spawn_blocking just hangs forever, because all your blocking threads are waiting on a pair of idle pipes (stdout and stderr of a child process, in our case)

Tokio PR tokio-rs/tokio#4824 explains why Windows child stdio was moved to the blocking pool.

Minimal deterministic reproduction

I added a Windows-only codex-rmcp-client integration test with a deliberately small blocking pool:

#[test]
fn idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool() -> anyhow::Result<()> {
    let runtime = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(2)
        .max_blocking_threads(8)
        .enable_all()
        .build()?;

    let sentinel_completed = runtime.block_on(async {
        let launcher = Arc::new(LocalStdioServerLauncher::new(std::env::current_dir()?));
        let mut clients = Vec::new();

        // Alternate Legacy and V20260728 transports. Each initialized server then
        // waits quietly on stdout while Codex independently drains quiet stderr.
        for index in 0..4 {
            let mode = if index % 2 == 0 {
                McpProtocolMode::Legacy
            } else {
                McpProtocolMode::V20260728
            };
            let env = (mode == McpProtocolMode::V20260728).then(|| HashMap::from([(
                OsString::from("CODEX_MCP_PROTOCOL_VERSION"),
                OsString::from("2026-07-28"),
            )]));
            let client = Arc::new(RmcpClient::new_stdio_client_with_protocol_mode(
                stdio_server_bin()?.into(), vec![], env, &[], None,
                launcher.clone(), mode,
            ).await?);
            client.initialize(init_params(), Some(Duration::from_secs(5)), no_elicitation()).await?;
            clients.push(client);
        }

        tokio::time::sleep(Duration::from_millis(250)).await;
        let sentinel = tokio::task::spawn_blocking(|| ());
        let completed = tokio::time::timeout(Duration::from_secs(1), sentinel).await.is_ok();

        // Release the pipe reads before returning so even the pre-fix runtime exits.
        for client in clients { client.shutdown().await; }
        Ok::<_, anyhow::Error>(completed)
    })?;

    assert!(sentinel_completed, "idle MCP stdio exhausted Tokio's blocking pool");
    Ok(())
}

Reproduction result

On exact upstream 042534ec1ab2:

TRY 1 FAIL [1.313s] idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool
TRY 2 FAIL [1.325s] idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool
idle MCP stdout/stderr reads exhausted Tokio's blocking pool

The same test also failed on a direct child of that release before changing the MCP code. There is no diff in the runtime/local-stdio files between those two revisions.

Proof-of-fix result

I prototyped a Windows-only transport replacement:

create three unique byte-mode named-pipe pairs per local MCP child;
keep parent endpoints as Tokio NamedPipeServer handles opened with FILE_FLAG_OVERLAPPED (IOCP);
pass synchronous client endpoints to the child as stdin/stdout/stderr;
retain existing Job Object containment and shutdown behavior;
leave Unix/macOS unchanged.

With that change, the identical test passes:

PASS [0.299s] idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool

The existing local STDIO tests also pass for both legacy and modern protocol modes, including the Windows Job Object descendant-cleanup test.

Expected behavior

Idle local STDIO MCP servers must not consume the finite shared blocking pool. Long-running app-server operation should preserve capacity for unrelated blocking operations regardless of how many retained MCP sessions exist.

Actual behavior

Approximately two permanently pending blocking reads are created per quiet local STDIO MCP client. With enough retained clients, later spawn_blocking tasks queue indefinitely. At the default limit this is reachable around 256 local clients, or earlier because the pool is shared with other blocking work.

Why raising `max_blocking_threads` is not sufficient

Increasing the limit only moves the failure threshold, permits more OS threads, and leaves an unbounded resource relationship. Disabling stderr capture frees only one slot per server and can deadlock verbose children on a full stderr pipe.

Related Codex issues/PRs

[Windows] Completed subagents leave STDIO MCP Node.js processes running (245 processes / 18 GB) #34658 — completed subagents retain large per-subagent STDIO MCP fleets
[Windows Desktop 26.810.7004] Thread-scoped STDIO MCP processes and local-proxy CLOSE_WAIT sockets accumulate #38981 — per-thread STDIO MCP process sets accumulate in long-lived Windows Desktop sessions
[Windows Codex app] Local stdio MCP servers are repeatedly spawned and not reaped within a single task #38754 / Windows/WSL2: Desktop retains one eager MCP stack per opened chat and replays history during navigation #32154 — repeated per-thread/turn MCP generations
Codex Desktop should use a project-scoped MCP process pool instead of starting MCP per session #20883 — proposal to share/pool MCP servers
Terminate stdio MCP servers on shutdown to avoid process leaks #19753 — explicit MCP shutdown at selected lifecycle boundaries (partial mitigation, not protection for many still-live transports)

I searched open and closed Codex issues/PRs for blocking pool, spawn_blocking, max_blocking_threads, ChildStdout, ChildStderr, and 512 threads; I did not find an existing Codex report for this specific mechanism.

Suggested fix

Use IOCP-compatible parent pipe endpoints on Windows (overlapped named pipes are one practical approach), plus retain deterministic process-tree shutdown. Independently, lazy MCP startup/shared bounded connection ownership would reduce process and memory amplification, but it should not be the only protection against exhausting a global runtime resource.

This is a very well-researched report — thank you for the failing regression test and the proof-of-fix. I independently verified the mechanism against the current code and it holds:

In codex-rs/arg0/src/lib.rs, build_runtime() (lines 290-295) builds a multi-thread Tokio runtime using enable_all() and thread_stack_size(...) but does NOT call max_blocking_threads(...), so Tokio uses its default of 512. Confirmed.
In codex-rs/rmcp-client/src/local_child.rs (lines 23-25), every local MCP child is spawned with piped stdin/stdout/stderr.
In codex-rs/rmcp-client/src/stdio_server_launcher.rs (lines 354-360), a tokio::spawn wraps stderr in BufReader::lines() and loops. On Windows, ChildStderr/ChildStdout are io::blocking::Blocking, so the pending read pins a shared blocking worker for the lifetime of the pipe.
codex-rs/Cargo.lock pins tokio to 1.52.3 (confirmed), so the Windows child-stdio-on-blocking-pool behavior you reference (max_blocking_threads documentation should perhaps indicate that Tokio internally uses those too tokio-rs/tokio#5777, Fix wayland image paste error #4824) applies.

So the two-pending-reads-per-quiet-client model is accurate: roughly two blocking workers consumed per retained local STDIO MCP server, and at the default 512 limit this becomes reachable — or can starve unrelated spawn_blocking work well before that when the pool is shared.

On the fix, your IOCP-compatible parent pipe endpoints (overlapped named pipes with FILE_FLAG_OVERLAPPED) are the correct low-level approach: it removes the child stdio from the blocking pool entirely rather than just raising the threshold. I agree with your note that raising max_blocking_threads only moves the failure point and leaves the unbounded resource relationship in place; and that disabling stderr capture is not safe (it can deadlock verbose children on a full pipe).

One thing worth flagging for the fix review: the change is Windows-only (cfg(windows)), so the regression test needs to remain Windows-gated too, and the existing local-stdio tests (Legacy + V20260728 protocol modes, and the Windows Job Object descendant-cleanup test you mentioned) must still pass. The deterministic shutdown-before-return in the test is a nice touch to avoid the pre-fix runtime hanging.

This deserves maintainer attention — it is a real shared-runtime-resource exhaustion, and the report is reproducible and has a concrete fix. Happy to help review the PR or validate the cross-platform-safe guard once opened.

Adding a field data point that matches this exactly (Codex CLI 0.156.0, codex app-server on Windows 11, used as a long-running server for a local chat client).

The app-server stopped answering thread/start and turn/start (reads such as thread/loaded/list kept working) twice, on two separate server processes. Each time it had 256-257 direct children: 126-128 node_repl.exe plus the same number of CUA node.exe companions, with 136-143 loaded threads.

On a second, isolated instance we loaded sessions one at a time: with 127 helper pairs thread/start answered in 243 ms; with 128 pairs every thread/start and turn/start timed out (~30 s). 2 helpers x 2 pipes x 128 = 512, the default max_blocking_threads. Minidumps of both the stalled test instance and a stalled production process show 511-512 Tokio blocking workers waiting in NtWaitForSingleObject.

So on Windows the practical ceiling is about 128 loaded sessions when each session starts two local STDIO MCP servers. The overlapped-pipe fix proposed here would remove it; an explicit unload (#21889) or pooled MCP processes (#20883) would also keep long-running app-servers away from it.

Additional source-level reproduction and a private prototype against public commit 9b738582b:

A Tokio runtime with two blocking workers and four quiet child processes reproduces starvation when stdout and stderr use ordinary process pipes. A spawn_blocking sentinel times out.
Replacing all three parent endpoints with Tokio IOCP-backed byte-mode named pipes makes the same workload pass. The child endpoints remain synchronous file handles, preserving existing stdio server behavior.
The prototype connects each pair before spawning, uses unpredictable single-instance names, rejects remote pipe clients, and preserves the existing suspended Job Object assignment and fallback.
Pipe creation supplies a protected owner/SYSTEM DACL. A native read-back test checks the actual kernel object's access-control entries.
Focused native checks pass for stdin/stdout/stderr round trip, stdin EOF, and the ordinary-versus-overlapped blocking-pool comparison. The affected MCP client crate compiles.
All five existing native stdio protocol tests pass, covering legacy framing, modern local discovery and MRTR, modern executor behavior, and legacy elicitation fallback.

The protocol and child-lifecycle checks are kept separate from the pool reproduction. This is a private source prototype, not a shipped desktop fix. Increasing the blocking-pool limit would only raise the saturation threshold; the underlying quiet pipe reads need to stop occupying that pool.

No external code PR is being opened, in accordance with the repository's current contribution policy.

Additional measured Windows evidence on the newer bundled backend codex-cli 0.162.0-alpha.17.2 (Desktop package 26.1007.2314.0, internal app 26.1007.21434). The local OS probe reports Windows 10 Pro x64, 10.0.19045.

I investigated this with Codex assistance. In an isolated diagnostic home with 270 configured minimal MCP servers, the direct-stdio case reached 567 backend threads and 516 Windows UserRequest waits. config/read timed out at 8,125 ms and configRequirements/read at 8,062 ms. Those are probe deadlines/lower bounds, not completed request latencies.

With the diagnostic servers accessed through authenticated literal-loopback Streamable HTTP, the backend had 73 threads and four UserRequest waits; both reads completed in 16 ms. The gateway retained a separate native stdio child per session. This moved pipe-reader pressure out of the Codex backend; it did not pool native session state or eliminate process multiplication.

Backend private memory was 130.6 MiB in the stdio sample versus 149 MiB in the HTTP sample. These are backend-only snapshots, so I am not claiming reduced total memory use.

A separate actual-server test used 12 isolated ephemeral contexts � 21 custom MCP servers = 252 live sessions. It recorded zero transport errors, 67 backend threads, three UserRequest waits, config/read median 19.1 ms / p95 29.9 ms / max 30.4 ms, and 12 successful fixed native status reads. Owner exit left zero test sessions. No inference or native asset writes were performed. The 21 Desktop catalogs retained their original 427 tool definitions and filters.

I also found that a companion host used the same codex-mcp-client identity. Explicit authenticated owner attribution was needed so Desktop exit did not remove the companion host's sessions. The gateway keeps DELETE/owner cleanup, bounded idle cleanup with pending-request protection, authentication and request/session limits.

The measurements support the local transport-pressure finding in this issue. I have not established that all workspace-routing/remote-compaction errors or long-running rendered UI stalls have this cause.

Sanitized report, test outline and result data: https://gist.github.com/CrispyW0nton/ec298528829397c5389dd49fba193578

This is additional field evidence and a local workaround, not an upstream fix or a claim of faster model generation. The bundle does not include a turnkey harness or modified Desktop binaries.

Windows Desktop confirmation

On Desktop 26.1013.4894.0 with bundled codex-cli 0.163.0-alpha.10.2, several open chats repeatedly led to thread/read, config/read and tool-discovery timeouts while CPU and disk activity stayed low. Restarting helped temporarily.

One backend reached 556 OS threads, including 500 Windows UserRequest waits. These counts support resource-pressure investigation but do not identify individual pipe-reader stacks.

Reproduction

The adjacent public source reproduces this at the real RmcpClient/LocalStdioServerLauncher boundary:

Use two async workers and an eight-worker blocking pool.
Three MCP clients complete echo calls. Starting a fourth quiet child fills the pool with pending stdout/stderr reads.
An unrelated spawn_blocking task misses its one-second deadline. With overlapped parent pipes, all four clients and the task complete.

The reduced pool accelerates the failure; four clients is not the normal product limit. The default blocking-pool limit is 512.

Retention amplifies the pressure

A separate control against the unchanged installed backend confirmed that an inactive, successfully unsubscribed context retained its quiet MCP process for 1,800.016 seconds before both were retired. That confirms 30-minute retention, rather than a permanent leak. Attribution to individual Desktop helper contexts remains unproven.

Scoping plugins and running tool-free reviews in isolated CLI processes reduced added MCP pressure, but shared Desktop timeouts later returned. A separate browser CDP timeout was also observed; this investigation does not attribute every stall to the stdio issue.

Fix

A local overlapped-pipe repair is implemented and tested. Implementation, verification and deployment limits are in a separate comment. The installed Desktop has not yet been switched to the candidate.

Local Windows fix implemented

The local source fix replaces Windows MCP parent stdio with overlapped named pipes. Quiet stdout/stderr reads no longer occupy Tokio's shared blocking workers. Children still receive synchronous standard handles through the existing suspended-spawn and Job Object launcher.

What changed

Parent pipe I/O uses Tokio/IOCP; the existing protocol parsers, frame limits, stderr handling and shutdown paths remain.
Pipes are first-instance, local-only and restricted to the current user. Parent handles are non-inheritable.
The non-Windows transport is unchanged.

Verification

At the real RmcpClient boundary, three clients completed echo calls before a fourth quiet child exhausted an eight-worker blocking pool. An unrelated blocking task then missed its one-second deadline. With the fix, all four clients and that task completed.

Fifteen focused checks passed, including protocol behavior, pipe security, EOF, cancellation, failed spawn and Job Object containment. Sixteen isolated cleanup repetitions left the process handle count unchanged. A P2 review found a test-isolation issue; it was fixed without changing the production implementation.

The complete Windows CLI builds. Its 452 exported JSON schema files match the installed backend byte-for-byte. Both binaries passed isolated initialization, config read and harmless command execution. The candidate also opened copies of the existing runtime databases without changing their schemas or migration histories.

Deployment status and limits

Active locally. The complete private bundle now runs in Desktop. After correcting the omitted companion executables, it passed an isolated test through the actual code host and command tool. The running app-server and code-host paths/hashes and selectors are verified; real shell execution and native chat reads pass. A bounded fresh-log check found no transport timeouts. Backups and selector rollback are retained.

This is a local candidate based on public commit cd643519fb56011557d09b41053a3a4c0e013e1f (0.163.0-alpha.10), adjacent to installed 0.163.0-alpha.10.2. Verification used Windows and Rust 1.99 rather than the repository-pinned 1.95. Sustained multi-chat recovery, restricted sandbox behavior and other platforms remain unverified.

No signed application file has been modified.

Two amplifiers on Desktop 26.1013.4894.0 / codex-cli 0.163.0-alpha.10.2

I reproduced the mechanism in this issue on a private codex app-server (0.163.0-alpha.10.2, scratch CODEX_HOME, one ephemeral thread, K copies of one small stdio MCP server):

	K=200	K=300
backend OS threads (peak)	458	558
`config/read` probes (1/s)	147/147, max 0.04 s	11 answered (all sent before saturation), 136 unanswered after 140 s
`thread/loaded/list` probes	147/147, max 0.004 s	147/147, max 0.008 s
MCP startups	200 ready	260 ready, 34 failed (startup timeout)

Two code paths make the pool fill sooner and explain why unrelated requests stall together, then flush together. I didn't see either in this thread yet:

An unscoped mcpServerStatus/list cold-starts every configured server. When serverName is absent, even with a threadId, list_mcp_server_status_response calls collect_mcp_server_status_snapshot_with_detail (codex-mcp/src/mcp/mod.rs). That builds a fresh McpConnectionSet with McpStartupPolicy::Eager for all effective servers, then cancels it. Each such call temporarily adds a full extra set of stdio pipe pairs on top of the per-thread sets. On a pool that is nearly full, one status poll is enough to tip it. For example, with 1 thread and 6 concurrent unscoped calls, I counted 63 server processes: 7 copies of a 9-server set.
Config loads are serialized behind a one-permit semaphore that needs spawn_blocking. refresh_application_network_policy() (app-server/src/application_network.rs) holds network_policy_reload, a Semaphore::new(1) (config_manager.rs). It holds it while reading local policy and the cloud-config cache, and the cache read uses spawn_blocking (cloud-config/src/cache.rs). Once the pool is full, the permit holder can't finish. So config/read, configRequirements/read, getAuthStatus, model/list and unscoped mcpServerStatus/list all queue behind it. On the live desktop that produced a 632 s config/read and a 465 s getAuthStatus, while 631 concurrent thread/read calls answered in under 0.2 s.

Suggested mitigations in addition to the named-pipe fix:

Answer an unscoped mcpServerStatus/list that has a threadId from the thread's live connections, not from a cold start.
Don't hold the policy-reload permit across blocking I/O.
Set max_blocking_threads explicitly until stdio no longer uses the blocking pool.

Local Windows observation after the pipe repair — 18 October 2026

The repaired backend answered every check during 125 minutes of normal-use observation (08:24–10:30 UTC), with the same process and no restart. Four scheduled checks, two extra checks and the baseline were completed.

Measure	First	Peak	Final
Backend OS threads	81	93	62
Direct helper processes	206	282	168
Backend working set	1.90 GiB	3.25 GiB	3.00 GiB

7/7 real command-tool checks passed; 24 native chat reads answered in 0.089–2.565 s.
No global backend stall, workspace-sync failure or code-host failure was detected. We checked 16,851 log lines across non-overlapping windows from 08:11–10:30 UTC. Existing app-discovery Auth required warnings continued and were classified separately.
One browser-control incident occurred: an Emulation.setFocusEmulationEnabled timeout followed by 30 frame-not-found errors. The affected chat continued command work; recovery of that specific browser operation was not verified.

The loaded segment included seven responding chats (six active turns and one completing during the sample); a concurrent backup was reported running. Later sampled turns completed and load subsided. Two source-preparation workers also contributed workload; compilation and synthetic load were excluded during the observation.

This is normal-use evidence, not a controlled A/B benchmark or proof of permanent recovery. Log reads were bounded; retained tails covered their requested windows. Restricted-sandbox behavior remains unverified. Connection-reuse and policy-reload follow-ups are separate drafts and have not been activated. The bounded watch is now paused.

Fix: An unfiltered mcpServerStatus/list with a threadId reuses matching live connections. Changed registrations still use current configuration.

Measured results — Windows, 0.163.0-alpha.10 with the pipe and status-reuse patches:

Repeated same-thread status requests: cumulative server starts 1 → 2 → 3 → 3 before, 1 → 1 → 1 → 1 after, using the same unchanged stdio fixture.
47 focused tests passed.
104 passive samples over 103 minutes; 3,747 logged responses, including errors. No recurrence of the full backend stop was detected.
Three responses took 51.7–58.1 seconds; the maximum over the following 65 minutes was 6.657 seconds.
Private memory: 1.75 → 2.71 GiB, peak 2.82 GiB.

Recommendation: retain the repair and investigate memory growth separately. The observation supports stability under this workload; it does not establish permanent recovery or a matched latency improvement.

Navigation Menu

Repository navigation

[Windows] Idle local STDIO MCP servers can exhaust Tokio blocking pool and stall the app-server #42740

What version of Codex are you using?

Platform

Summary

Confirmed mechanism

Minimal deterministic reproduction

Reproduction result

Proof-of-fix result

Expected behavior

Actual behavior

Why raising `max_blocking_threads` is not sufficient

Related Codex issues/PRs

Suggested fix

Windows Desktop confirmation

Reproduction

Retention amplifies the pressure

Fix

Local Windows fix implemented

What changed

Verification

Deployment status and limits

Two amplifiers on Desktop 26.1013.4894.0 / codex-cli 0.163.0-alpha.10.2

Local Windows observation after the pipe repair — 18 October 2026

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Windows] Idle local STDIO MCP servers can exhaust Tokio blocking pool and stall the app-server #42740

Description

What version of Codex are you using?

Platform

Summary

Confirmed mechanism

Minimal deterministic reproduction

Reproduction result

Proof-of-fix result

Expected behavior

Actual behavior

Why raising max_blocking_threads is not sufficient

Related Codex issues/PRs

Suggested fix

Activity

argszero commented on Sep 4, 2026

kevinwwillis commented on Sep 27, 2026

4Liberty commented on Oct 8, 2026

CrispyW0nton commented on Oct 12, 2026

K0enna commented on Oct 17, 2026

Windows Desktop confirmation

Reproduction

Retention amplifies the pressure

Fix

K0enna commented on Oct 18, 2026

Local Windows fix implemented

What changed

Verification

Deployment status and limits

PNWGMR commented on Oct 18, 2026

Two amplifiers on Desktop 26.1013.4894.0 / codex-cli 0.163.0-alpha.10.2

K0enna commented on Oct 18, 2026

Local Windows observation after the pipe repair — 18 October 2026

K0enna commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Why raising `max_blocking_threads` is not sufficient