app-server: app/installed rebuilds the codex_apps MCP session per call, pinning ~120% of a core while idle #38480

Summary

On a Linux host running codex app-server --listen unix:// for a Codex Desktop SSH-remote session, the app-server sat at 120–127% of one CPU core while completely idle (no active turn, no user present). The cause is that app/installed / app/list requests rebuild the entire codex_apps MCP session from scratch on every call, including a full system-CA-bundle re-parse and a fresh TLS handshake, and then drop the RunningService without close().

Restarting the app-server does not help — the Desktop client reconnects and resumes the same poll loop. Dropping the client connection (killing the codex app-server proxy relay) stops it immediately.

This is adjacent to #30606 but a different mechanism: no ambient-suggestion turns are involved here, and the hot path is MCP session construction rather than SQLite log churn.

Environment

• Remote: Debian, kernel 6.12.95, x86_64
• codex-cli 0.144.0, vendored musl build (codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex)
• Listener: codex -c features.code_mode_host=true app-server --listen unix://
• Client: Codex Desktop over SSH remote, app_server.client_name="Codex Desktop"
• Enabled bundled plugins: sites@openai-bundled, visualize@openai-bundled (these back the codex_apps MCP server, 45 tools)

Observed behaviour

The client repeatedly issued only these requests — no others — from a single connection:

app/installed          0.40/s
app/list               0.10/s
mcpServerStatus/list   0.02/s

Each one drove a complete codex_apps rebuild cycle:

1. New HTTPS client built per call, re-reading and re-parsing the system CA bundle:

   INFO codex_http_client::custom_ca  transport_worker{name="StreamableHttpClientWorker"}:
     using system root certificates because no CA override environment variable was selected
     codex_ca_certificate_configured=false ssl_cert_file_configured=false
   

   Measured at 4.7/s.

2. No connection reuse — the whole pool is torn down each cycle:

   TRACE hyper_util::client::legacy::pool  pool dropped, dropping pooled (("https", chatgpt.com))
   TRACE hyper_util::client::legacy::pool  pool closed, canceling idle interval
   TRACE hyper_util::client::legacy::pool  checkout waiting for idle connection: ("https", chatgpt.com)
   TRACE hyper_util::client::legacy::connect::http  Http::connect; scheme=Some("https"), host=Some("chatgpt.com")
   

3. Full initialize + ListResources + ListTools every time, with a ~29.5 KB ListResources payload serialized into a TRACE record:

   TRACE rmcp::service  new{server_name=codex_apps}:start_server_task{server_name=codex_apps}:initialize:serve_inner:
     new event evt=PeerMessage(Response(JsonRpcResponse { ... ListResourcesResult { ... resources: [...] } }))
   

4. 664 KB tool-schema cache rewritten each cycle at ~/.codex/cache/codex_apps_tools/<hash>.json.

5. RunningService dropped without close(), cancelling the transport:

   DEBUG rmcp::service  RunningService dropped without explicit close(). The connection will be
     closed asynchronously. For guaranteed cleanup, call close() or cancel() before dropping.
   INFO  rmcp::service  ...:serve_inner: serve finished quit_reason=Cancelled
   INFO  rmcp::service  ...:serve_inner: task cancelled
   DEBUG rmcp::transport::worker  worker quit with reason: Cancelled
   DEBUG rmcp::transport::streamable_http_client  transport_worker{name="StreamableHttpClientWorker"}: cancelled
   

6. The tool cache is never used. On every single pass:

   TRACE codex_mcp::connection_manager  list_all_tools{mcp_server_count=1}: waiting for MCP server tools
     while building tool list server_name=codex_apps has_cached_tools=true startup_complete=false
   TRACE codex_mcp::connection_manager  list_all_tools{mcp_server_count=1}: listed MCP server tools
     while building tool list server_name=codex_apps tool_count=45
   

   startup_complete=false never latches true despite has_cached_tools=true, so the fast path is dead code in this state.

Notably there were zero WARN and zero ERROR log entries throughout. The transport is not failing on the network — Codex cancels it itself, every cycle.

Measurements

Taken on the live process, with care not to attribute diagnostic overhead to the workload.

Load was spread evenly across all 8 tokio-rt-worker threads (work stealing), which is why no single thread looked hot in top -H and why this is easy to misread as a runtime-level spin rather than a repeated workload. The sqlx-sqlite-worker threads were cold (~70 jiffies total), ruling out query execution as the hot path.

A healthy reconnected client costs a single ~5 s burst peaking at 237% during handshake, then settles to 0%. So client presence is not the problem — only the stuck poll loop is.

Reproduction

1. Run codex app-server --listen unix:// on a Linux remote with the bundled sites / visualize plugins enabled.
2. Connect Codex Desktop to it over SSH remote.
3. Leave it idle. When the client enters the app/installed poll loop, the app-server pins ~120% of a core indefinitely.
4. kill -TERM the codex app-server proxy relay → CPU drops to 0% immediately. Restarting the app-server instead does not help, because the client reconnects into the same loop.

Suggested fixes

1. Reuse a pooled HTTPS client instead of constructing one per MCP session. Rebuilding the rustls root store from the system CA bundle 4.7 times a second is the single largest cost and is pure waste — the trust anchors do not change between calls.
2. Latch startup_complete so has_cached_tools=true actually short-circuits the rebuild in list_all_tools. As written, the cache is populated and then ignored.
3. Call close() on RunningService rather than dropping it — rmcp explicitly warns about this, and the cancel/respawn churn is what multiplies the cost across the Tokio workers.
4. Consider rate-limiting or coalescing app/installed / app/list, so a misbehaving client cannot amplify one poll into a full remote MCP handshake.

Side observation

~/.codex/logs_2.sqlite had grown to 1.76 GB with only 0.37 GB live data — 309,706 of 461,104 pages were freelist (~1.18 GB, 67%, reclaimable). Retention deletes rows but never VACUUMs, so the file only grows. Not the cause of the CPU issue, but it makes every log insert progressively more expensive and is worth a periodic VACUUM or auto_vacuum=INCREMENTAL.