[Windows] Idle local STDIO MCP servers can exhaust Tokio blocking pool and stall the app-server

What version of Codex are you using?

Reproduced from upstream commit 042534ec1ab2f79c2997e779347d5383832ecb2e (Release 0.154.0-alpha.1). The same local-stdio implementation is still present on current main at the time of filing.

Platform

Windows 11 x64.

Summary

On Windows, every quiet local STDIO MCP client can occupy two workers in Tokio's shared blocking pool indefinitely: one pending read for child stdout and one for child stderr. Codex uses Tokio's default blocking-thread limit (512), so enough retained/per-thread MCP clients can prevent unrelated spawn_blocking work from ever starting and stall the entire app-server even though async network I/O can continue.

This is a separate low-level failure mode underneath the already reported MCP process/session multiplication issues. Explicit shutdown helps only after an owner is actually disposed; it does not make hundreds of simultaneously retained local STDIO transports safe on Windows.

Confirmed mechanism

Codex currently does the following:

1. codex-rs/arg0/src/lib.rs::build_runtime() creates a multi-thread Tokio runtime without setting max_blocking_threads (Tokio default: 512).
2. codex-rs/rmcp-client/src/local_child.rs starts every local MCP child with piped stdin/stdout/stderr.
3. The rmcp transport continuously reads stdout.
4. codex-rs/rmcp-client/src/stdio_server_launcher.rs starts a second continuous BufReader::lines() drain for stderr.
5. Tokio 1.52.3 represents Windows ChildStdout/ChildStderr as io::blocking::Blocking<ArcFile>, so each pending read consumes a shared blocking worker.

Tokio documents the exact generic failure in tokio-rs/tokio#5777:

on Windows, reading a pipe is a blocking read and will take away a blocking thread forever

and:

spawn_blocking just hangs forever, because all your blocking threads are waiting on a pair of idle pipes (stdout and stderr of a child process, in our case)

Tokio PR tokio-rs/tokio#4824 explains why Windows child stdio was moved to the blocking pool.

Minimal deterministic reproduction

I added a Windows-only codex-rmcp-client integration test with a deliberately small blocking pool:

#[test]
fn idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool() -> anyhow::Result<()> {
    let runtime = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(2)
        .max_blocking_threads(8)
        .enable_all()
        .build()?;

    let sentinel_completed = runtime.block_on(async {
        let launcher = Arc::new(LocalStdioServerLauncher::new(std::env::current_dir()?));
        let mut clients = Vec::new();

        // Alternate Legacy and V20260728 transports. Each initialized server then
        // waits quietly on stdout while Codex independently drains quiet stderr.
        for index in 0..4 {
            let mode = if index % 2 == 0 {
                McpProtocolMode::Legacy
            } else {
                McpProtocolMode::V20260728
            };
            let env = (mode == McpProtocolMode::V20260728).then(|| HashMap::from([(
                OsString::from("CODEX_MCP_PROTOCOL_VERSION"),
                OsString::from("2026-07-28"),
            )]));
            let client = Arc::new(RmcpClient::new_stdio_client_with_protocol_mode(
                stdio_server_bin()?.into(), vec![], env, &[], None,
                launcher.clone(), mode,
            ).await?);
            client.initialize(init_params(), Some(Duration::from_secs(5)), no_elicitation()).await?;
            clients.push(client);
        }

        tokio::time::sleep(Duration::from_millis(250)).await;
        let sentinel = tokio::task::spawn_blocking(|| ());
        let completed = tokio::time::timeout(Duration::from_secs(1), sentinel).await.is_ok();

        // Release the pipe reads before returning so even the pre-fix runtime exits.
        for client in clients { client.shutdown().await; }
        Ok::<_, anyhow::Error>(completed)
    })?;

    assert!(sentinel_completed, "idle MCP stdio exhausted Tokio's blocking pool");
    Ok(())
}

Reproduction result

On exact upstream 042534ec1ab2:

TRY 1 FAIL [1.313s] idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool
TRY 2 FAIL [1.325s] idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool
idle MCP stdout/stderr reads exhausted Tokio's blocking pool

The same test also failed on a direct child of that release before changing the MCP code. There is no diff in the runtime/local-stdio files between those two revisions.

Proof-of-fix result

I prototyped a Windows-only transport replacement:

• create three unique byte-mode named-pipe pairs per local MCP child;
• keep parent endpoints as Tokio NamedPipeServer handles opened with FILE_FLAG_OVERLAPPED (IOCP);
• pass synchronous client endpoints to the child as stdin/stdout/stderr;
• retain existing Job Object containment and shutdown behavior;
• leave Unix/macOS unchanged.

With that change, the identical test passes:

PASS [0.299s] idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool

The existing local STDIO tests also pass for both legacy and modern protocol modes, including the Windows Job Object descendant-cleanup test.

Expected behavior

Idle local STDIO MCP servers must not consume the finite shared blocking pool. Long-running app-server operation should preserve capacity for unrelated blocking operations regardless of how many retained MCP sessions exist.

Actual behavior

Approximately two permanently pending blocking reads are created per quiet local STDIO MCP client. With enough retained clients, later spawn_blocking tasks queue indefinitely. At the default limit this is reachable around 256 local clients, or earlier because the pool is shared with other blocking work.

Why raising max_blocking_threads is not sufficient

Increasing the limit only moves the failure threshold, permits more OS threads, and leaves an unbounded resource relationship. Disabling stderr capture frees only one slot per server and can deadlock verbose children on a full stderr pipe.

Related Codex issues/PRs

• #34658 — completed subagents retain large per-subagent STDIO MCP fleets
• #38981 — per-thread STDIO MCP process sets accumulate in long-lived Windows Desktop sessions
• #38754 / #32154 — repeated per-thread/turn MCP generations
• #20883 — proposal to share/pool MCP servers
• #19753 — explicit MCP shutdown at selected lifecycle boundaries (partial mitigation, not protection for many still-live transports)

I searched open and closed Codex issues/PRs for blocking pool, spawn_blocking, max_blocking_threads, ChildStdout, ChildStderr, and 512 threads; I did not find an existing Codex report for this specific mechanism.

Suggested fix

Use IOCP-compatible parent pipe endpoints on Windows (overlapped named pipes are one practical approach), plus retain deterministic process-tree shutdown. Independently, lazy MCP startup/shared bounded connection ownership would reduce process and memory amplification, but it should not be the only protection against exhausting a global runtime resource.

Start with codex-rs/arg0/src/lib.rs::build_runtime(), codex-rs/rmcp-client/src/local_child.rs, and codex-rs/rmcp-client/src/stdio_server_launcher.rs to trace runtime limits and Windows stdout/stderr reads. Run the Windows-only codex-rmcp-client integration test idle_mcp_stdio_does_not_exhaust_tokio_blocking_pool. Done means idle local STDIO MCP servers no longer consume the shared blocking pool and existing local STDIO and Job Object cleanup tests still pass.