Responses WebSocket black-hole after VPN disconnect leaves CLI Thinking for 5 minutes #42914

Environment

What issue are you seeing?

The original symptom was disconnecting a VPN while an active Codex CLI turn was still shown as Thinking. The CLI then appeared to think indefinitely (at least several minutes) instead of detecting the loss and reconnecting. Killing Codex and starting it again after connectivity was restored worked immediately.

I reproduced the underlying failure with a controlled network black hole. A local CONNECT proxy allowed the authenticated Responses WebSocket to establish, then stopped forwarding bytes in both directions while deliberately retaining both TCP connections. This simulates a VPN/path-routing loss where no FIN or RST reaches the client.

With the normal default configuration, Codex remained in the active turn for 299.987 seconds after traffic stopped. At 15 seconds into the stall, ss still reported the Codex-to-proxy and proxy-to-upstream connections as ESTABLISHED with empty queues. The first error was:

stream disconnected before completion: idle timeout waiting for websocket

The first retry began immediately at that point, but the release UI did not display it. The replacement WebSocket handshake then consumed its 15-second connection timeout, and only retry two showed:

Reconnecting... 2/5 (request timed out)

Starting a new direct Codex turn after stopping the black-hole proxy completed in about five seconds. This explains why restart appears to fix the original VPN scenario: it abandons the wedged socket rather than waiting for its passive read timeout.

Steps to reproduce the bug

1. Start an authenticated Codex CLI turn through the Responses WebSocket path and wait until it is active/Thinking.
2. While keeping the existing TCP connection open, remove the route that carries its traffic. A VPN disconnect is one real-world trigger. For a deterministic test, use a CONNECT proxy that forwards the initial TLS/WebSocket traffic, then drops all subsequent bytes in both directions without closing either socket.
3. Observe the active turn and the client socket.

Controlled timeline (UTC):

As a control, configuring a custom provider with stream_idle_timeout_ms = 12000 changed detection from 299.987 seconds to 11.990 seconds. This confirms the default stream idle timeout is what gates detection. Lowering it globally is not a safe fix because healthy reasoning can produce no response frames for that length of time.

Expected behavior

Codex should detect that an established Responses WebSocket is no longer live promptly, show a reconnecting status at the first retry, and preserve legitimate long-running reasoning that has no user-visible output.

Investigation

• codex-rs/model-provider-info/src/lib.rs sets DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000.
• run_websocket_response_stream in codex-rs/codex-api/src/endpoint/responses_websocket.rs passively wraps ws_stream.next() in that timeout.
• The Responses WebSocket pump replies to a server Ping with Pong, but it does not originate client Ping frames or require a Pong deadline.
• In this failure, the timeout is ApiError::Stream("idle timeout waiting for websocket"), mapped by api_bridge.rs to CodexErr::Stream.
• The unbounded retry branch in core/src/responses_retry.rs applies only to CodexErrorDetails::ConnectionFailed, so this takes the ordinary bounded stream-retry path. The unbounded-retry feature is not the source of the five-minute detection delay.
• Retry one is intentionally hidden in the release UI; this makes the stall look worse but is secondary to transport liveness detection.

codex-rs/app-server-transport/src/transport/remote_control/websocket.rs already has the relevant pattern: periodic client Ping plus a bounded Pong deadline.

Proposed direction and tested prototype

I have a small local prototype against responses_websocket.rs (166 additions, 4 deletions). It:

1. sends a WebSocket Ping every 15 seconds;
2. fails the connection if no Pong arrives within 45 seconds;
3. retains the independent 300-second response-data idle timeout, so Pongs prove transport liveness but do not make a silent model response run forever.

Two local raw-WebSocket tests passed:

mise exec rust@1.95.0 -- just test -p codex-api websocket_liveness_watchdog
# 2 passed, 182 skipped

They verify (a) an unanswered Ping returns websocket pong timeout, and (b) a silent peer that replies to Ping reaches the existing response-data idle timeout instead of being killed by the watchdog. I can turn this prototype into a PR if maintainers agree with the approach.

Diagnostics

codex doctor --json was healthy before reproduction. Relevant redacted results:

{
  "overallStatus": "ok",
  "codexVersion": "0.153.4",
  "auth": "ChatGPT configured",
  "model": "gpt-5.6-terra",
  "provider": "openai",
  "network.websocket_reachability": "Responses WebSocket handshake succeeded",
  "websocket_connect_timeout_ms": 15000,
  "wire_api": "responses"
}

Raw traces are available locally but include authentication-related HTTP headers, so I have not attached them publicly. Safe timestamps and socket-state evidence are summarized above.

Related: #20739 (WebSocket closes before response.completed) and #24260 (long visible stall). This is distinct from #18471, which appears to be a Desktop conversation-state race.