Skip to content

codex-cli stalls for exactly 300s between tool result and next model request (stream disconnected) #23807

Open

Description

@dehnck

What version of Codex CLI is running?

codex-cli 0.132.0

What subscription do you have?

Plus

Which model were you using?

gpt-5.5

What platform is your computer?

Microsoft Windows NT 10.0.26100.0 x64

What terminal emulator and version are you using (if applicable)?

VS Code integrated terminal (PowerShell), codex running via openai.chatgpt VS Code extension

Codex doctor report

{
  "schemaVersion": 1,
  "generatedAt": "1779332610s since unix epoch",
  "overallStatus": "fail",
  "codexVersion": "0.132.0",
  "checks": {
    "app_server.status": {
      "id": "app_server.status",
      "category": "app-server",
      "status": "ok",
      "summary": "background server is not running",
      "details": {
        "control socket": "C:\\Users\\zhouj\\.codex\\app-server-control\\app-server-control.sock",
        "daemon state dir": "C:\\Users\\zhouj\\.codex\\app-server-daemon",
        "mode": "ephemeral",
        "pid file": "C:\\Users\\zhouj\\.codex\\app-server-daemon\\app-server.pid (missing)",
        "settings": "C:\\Users\\zhouj\\.codex\\app-server-daemon\\settings.json (missing)",
        "status": "not running",
        "update-loop pid file": "C:\\Users\\zhouj\\.codex\\app-server-daemon\\app-server-updater.pid (missing)"
      },
      "remediation": null,
      "durationMs": 0
    },
    "auth.credentials": {
      "id": "auth.credentials",
      "category": "auth",
      "status": "ok",
      "summary": "auth is configured",
      "details": {
        "auth file": "C:\\Users\\zhouj\\.codex\\auth.json",
        "auth storage mode": "File",
        "stored API key": "true",
        "stored ChatGPT tokens": "false",
        "stored agent identity": "false",
        "stored auth mode": "api_key"
      },
      "remediation": null,
      "durationMs": 0
    },
    "config.load": {
      "id": "config.load",
      "category": "config",
      "status": "ok",
      "summary": "config loaded",
      "details": {
        "CODEX_HOME": "C:\\Users\\zhouj\\.codex",
        "config.toml": "C:\\Users\\zhouj\\.codex\\config.toml",
        "config.toml parse": "ok",
        "enabled feature flags": "shell_tool, shell_snapshot, terminal_resize_reflow, sqlite, hooks, enable_request_compression, multi_agent, apps, tool_search, tool_suggest, plugins, plugin_hooks, in_app_browser, browser_use, browser_use_external, computer_use, plugin_sharing, image_generation, skill_mcp_dependency_install, steer, guardian_approval, goals, collaboration_modes, tool_call_mcp_elicitation, personality, fast_mode, tui_app_server, workspace_dependencies",
        "feature flag overrides": "goals=true",
        "feature flags enabled": "28",
        "log dir": "C:\\Users\\zhouj\\.codex\\log",
        "mcp servers": "0",
        "model": "gpt-5.5",
        "model provider": "custom",
        "sqlite home": "C:\\Users\\zhouj\\.codex"
      },
      "remediation": null,
      "durationMs": 0
    },
    "installation": {
      "id": "installation",
      "category": "install",
      "status": "ok",
      "summary": "installation looks consistent",
      "details": {
        "PATH codex #1": "C:\\Users\\zhouj\\AppData\\Roaming\\npm\\codex",
        "PATH codex #2": "C:\\Users\\zhouj\\AppData\\Roaming\\npm\\codex.cmd",
        "PATH codex entries": "2",
        "current executable": "C:\\Users\\zhouj\\AppData\\Roaming\\npm\\node_modules\\@openai\\codex\\node_modules\\@openai\\codex-win32-x64\\vendor\\x86_64-pc-windows-msvc\\codex\\codex.exe",
        "install context": "npm",
        "managed by bun": "false",
        "managed by npm": "true",
        "managed package root": "C:\\Users\\zhouj\\AppData\\Roaming\\npm\\node_modules\\@openai\\codex",
        "npm update target": "C:\\Users\\zhouj\\AppData\\Roaming\\npm\\node_modules\\@openai\\codex"
      },
      "remediation": null,
      "durationMs": 358
    },
    "mcp.config": {
      "id": "mcp.config",
      "category": "mcp",
      "status": "ok",
      "summary": "no MCP servers configured",
      "details": {},
      "remediation": null,
      "durationMs": 0
    },
    "network.env": {
      "id": "network.env",
      "category": "network",
      "status": "ok",
      "summary": "network-related environment looks readable",
      "details": {
        "proxy env vars present": "NO_PROXY, no_proxy"
      },
      "remediation": null,
      "durationMs": 0
    }
  }
}

What issue are you seeing?

Codex CLI periodically stalls for exactly 300 seconds (5 minutes) during a turn, showing "Working (Xm Ys)" with no network activity. After the 300s timeout, it logs stream disconnected - retrying sampling request (1/5) and resumes normally. This happens multiple times per session (5 occurrences across ~15 hours), making long-running agentic tasks unreliable.

From TUI log (~/.codex/log/codex-tui.log):

Instance 1 — exact 300.003s gap:
2026-05-20T12:21:44.542667Z INFO codex_core::client: close time.busy=10.9ms time.idle=40.2s
*** 300 seconds of silence — no client:new, no ToolCall ***
2026-05-20T12:26:44.545840Z WARN codex_core::session::turn: stream disconnected - retrying sampling request (1/5 in 197ms)...
2026-05-20T12:26:44.746956Z INFO codex_core::client: new <- retry succeeds immediately

Instance 2 — exact 300.021s gap:
2026-05-20T13:12:50.025022Z INFO codex_core::client: close time.busy=25.9ms time.idle=58.0s
*** 300 seconds of silence ***
2026-05-20T13:17:50.046571Z WARN codex_core::session::turn: stream disconnected - retrying sampling request (1/5 in 218ms)...
2026-05-20T13:17:50.267074Z INFO codex_core::client: new <- retry succeeds immediately

All 5 occurrences in this session:

  • 2026-05-20T12:26:44 (retry in 197ms)
  • 2026-05-20T13:17:50 (retry in 218ms)
  • 2026-05-20T15:42:53 (retry in 210ms)
  • 2026-05-20T15:52:04 (retry in 180ms)
  • 2026-05-21T00:56:48 (retry in 196ms)

The retry always succeeds in under 250ms, proving the network path is healthy. The stall is entirely internal to codex_core.

Additional context:

  • app-server event consumer lagged; dropping ignored events skipped=93/135/231 appears in the same session
  • The same local proxy (127.0.0.1:15721) handles Claude Code traffic without any stalls
  • Average time.idle across all 583 requests in this session is 85s (normal model thinking time)
  • Large time.idle values up to 1262s occur during active HTTP streams WITHOUT triggering "stream disconnected", confirming the 300s watchdog only fires between requests

What steps can reproduce the bug?

  1. Configure codex with a custom provider endpoint:

    model_provider = "custom"
    model = "gpt-5.5"
    model_reasoning_effort = "xhigh"
    sandbox_mode = "danger-full-access"
    
    [model_providers.custom]
    name = "custom"
    wire_api = "responses"
    supports_websockets = false
    requires_openai_auth = true
    base_url = "http://127.0.0.1:15721/v1"
    
  2. Start a codex session with a complex task that requires multiple tool calls (file reads, shell commands, code edits)

  3. Let it run for 30+ minutes

  4. Observe periodic 300-second gaps where no HTTP requests are sent to the model endpoint, followed by "stream disconnected - retrying sampling request (1/5)"

Detailed sequence before first stall:

12:20:40 client: close (model returned tool_call response, time.idle=29.5s)
12:20:40 ToolCall: shell_command (read test_supervisor.py)
12:20:40 ToolCall: shell_command (read test_session_adapter.py)
12:20:40 ToolCall: shell_command (git diff)
12:21:03 client: close (model responded, time.idle=22.4s)
12:21:04 ToolCall: shell_command (Get-Content SKILL.md) <- executes in ~200ms
12:21:04 client: new <- sends tool result back to model
12:21:44 client: close time.idle=40.2s <- model responds normally
*** 300 seconds of silence — no client:new, no ToolCall ***
12:26:44 stream disconnected - retrying (1/5 in 197ms)
12:26:44 client: new <- retry works instantly
The model's response at 12:21:44 completes normally. After client: close, codex_core should either issue the next client: new or complete the turn. Instead, nothing happens for exactly 300 seconds.

No thread ID available (custom provider endpoint).

What is the expected behavior?

After client: close, the turn should immediately either:

  1. Dispatch the next tool call from the response, or
  2. Send the next client: new request with tool results, or
  3. Complete the turn

There should be no 300-second idle gap between a successful response and the next action. The 300s value is suspiciously precise (300.003s and 300.021s), suggesting a hardcoded timeout constant in codex_core::session::turn that fires when the turn state machine fails to advance after receiving a complete response.

Additional information

Analysis:

The 300-second value appears to be a hardcoded stream-activity watchdog in codex_core::session::turn. The pattern suggests a race condition where the turn state machine sometimes fails to advance after receiving a complete model response. It sits idle until the watchdog fires and forces a retry.

This is NOT a network issue because:

  • The same local proxy handles Claude Code traffic without any stalls
  • The retry after 300s succeeds in <250ms every time
  • time.idle values on normal requests (22-134s) show the model endpoint is responsive

Possibly related:

  • app-server event consumer lagged; dropping ignored events skipped=93/135/231 — TUI event loop under pressure, may indicate backpressure blocking the turn from advancing
  • Multiple codex_core::tools::router: error=Exit code: 1 errors in the same session (tool execution failures)
  • model_reasoning_effort = "xhigh" produces longer model responses and more tool calls per turn, possibly increasing the chance of hitting the race condition

Suggested investigation areas:

  1. The 300s constant in codex_core::session::turn — is this a stream keepalive/watchdog timer?
  2. Possible race condition between the response parser completing and the turn state machine's next-step dispatch
  3. Whether app-server event consumer lagged indicates backpressure that blocks the turn from advancing
  4. Whether the responses wire API path has a different state machine than the chat path

Log grep patterns for reproduction:

grep "stream disconnected" ~/.codex/log/codex-tui.log
grep "client: close\|client: new" ~/.codex/log/codex-tui.log
grep "app-server event consumer lagged" ~/.codex/log/codex-tui.log
Session stats:

Log spans: 2026-05-05 to 2026-05-21
Total client new/close pairs: 1255
Stream disconnects: 5
Average time.idle: 85s
Max time.idle (without triggering disconnect): 5463s

Activity

  1. added
    CLIIssues related to the Codex CLI
    windows-osIssues related to Codex on Windows systems
    custom-modelIssues related to custom model providers (including local models)
    connectivityIssues involving networking or endpoint connectivity problems (disconnections)
    on May 21, 2026
  2. github-actions commented on May 21, 2026

    @github-actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  3. jshaofa-ui commented on May 21, 2026

    @jshaofa-ui

    Summary

    Issue: codex-cli 0.132.0 stalls for exactly 300 seconds (5 minutes) between completing a tool execution and sending the next model request. This occurs on Windows with VS Code integrated terminal (PowerShell), using gpt-5.5 model with Plus subscription.

    Root Cause: The Responses API WebSocket connection is reused across multiple model requests within a single turn. When the WebSocket connection becomes stale or dead (due to network issues, proxy timeouts, server-side disconnection, or Windows-specific networking behavior), the client does not detect this until it sends the next request and waits for the first event. The stream_idle_timeout default of 300,000ms (300s = 5 minutes) is used as the timeout for waiting for the first event after sending a request, causing the exact 300-second stall observed.

    Fix Approach: Introduce a separate first_event_timeout (default: 60s) for waiting on the first event after sending a WebSocket request, distinct from the existing stream_idle_timeout (default: 300s) used for subsequent events during streaming. Add WebSocket connection health detection via ping/pong keepalive. This ensures dead connections are detected within 60s rather than 300s, with automatic reconnection on the next request attempt.


    Root Cause Analysis

    Architecture Overview

    The codex-cli turn execution follows this flow:

    run_turn() {
        loop {
            run_sampling_request() {
                client_session.stream() {
                    // WebSocket path (preferred):
                    stream_responses_websocket() {
                        // Reuses existing WebSocket connection
                        websocket_connection.stream_request() {
                            send_websocket_request()     // Send request frame
                            run_websocket_response_stream() {
                                loop {
                                    timeout(idle_timeout, ws_stream.next())  // Wait for events
                                    // Process SSE events...
                                }
                            }
                        }
                    }
                    // HTTP path (fallback):
                    stream_responses_api() {
                        // Creates NEW HTTP connection per request
                    }
                }
                // Process stream events (tool calls, text deltas, etc.)
                // When tool call found: spawn tool execution future
                // When response.completed: break stream loop
            }
            drain_in_flight()  // Wait for tool execution to complete
            sess.record_conversation_items()  // Record tool results
            // Loop continues → next model request
        }
    }
    

    The 300-Second Timeout

    The default stream idle timeout is defined in model-provider-info/src/lib.rs:

    const DEFAULT_STREAM_IDLE_TIMEOUT_MS: u64 = 300_000;  // 300 seconds = 5 minutes

    This timeout is applied in two places in the WebSocket path:

    1. send_websocket_request() - Timeout for sending the request frame:
    // codex-api/src/endpoint/responses_websocket.rs
    async fn send_websocket_request(...) {
        let result = tokio::time::timeout(
            idle_timeout,  // 300s default
            ws_stream.send(Message::Text(request_text.into())),
        ).await...
    }
    1. run_websocket_response_stream() - Timeout for waiting for each event:
    // codex-api/src/endpoint/responses_websocket.rs
    async fn run_websocket_response_stream(...) {
        send_websocket_request(...).await?;  // Uses idle_timeout
    
        loop {
            let response = tokio::time::timeout(
                idle_timeout,  // 300s default - used for FIRST event AND subsequent events
                ws_stream.next()
            ).await
            .map_err(|_| ApiError::Stream("idle timeout waiting for websocket".into()));
            // ...
        }
    }

    Why the Stall Occurs

    The stall sequence:

    1. Model request 1 → Tool call detected → Tool executed → Result recorded
    2. Turn loop continues → New run_sampling_request() called
    3. WebSocket connection reused → stream_request() called on existing connection
    4. Connection is dead/stale → Server has closed it, or network path is broken
    5. send_websocket_request() → TCP send may succeed (data buffered locally) or fail silently
    6. run_websocket_response_stream() → Waits for first event with 300s timeout
    7. No events arrive (connection is dead) → 300s passes → Idle timeout fires
    8. Error propagates → CodexErr::Stream("idle timeout waiting for websocket", None)
    9. Turn fails with error event

    Why Windows/VS Code is Affed

    Several factors make this more likely on Windows with VS Code integrated terminal:

    1. TCP Keepalive Differences: Windows TCP keepalive defaults (2 hours) are much longer than Linux (75 seconds), so dead connections aren't detected at the TCP level.

    2. Proxy/Load Balancer Behavior: Corporate networks on Windows often route through proxies that may silently drop idle WebSocket connections after a timeout period.

    3. VS Code Terminal Multiplexing: The integrated terminal adds an additional layer of I/O handling that can mask connection state changes.

    4. PowerShell I/O Handling: PowerShell's handling of stdout/stderr streams may buffer or delay error propagation.

    5. Network Profile Changes: Windows network profile changes (WiFi ↔ Ethernet, VPN connect/disconnect) can break connections without proper TCP FIN/RST.

    Why HTTP Path Doesn't Exhibit This Issue

    The HTTP/SSE path creates a new connection for each request, so a dead connection is detected immediately during TCP connect (connection refused), not after waiting 300s. The WebSocket path reuses the same connection, so dead connections are only detected when data is sent and no response arrives.


    Proposed Fix

    Approach

    The fix has three components:

    1. Separate First-Event Timeout: Introduce first_event_timeout (default: 60s) for waiting on the first event after sending a WebSocket request. This is distinct from stream_idle_timeout (300s) which remains for subsequent events during streaming.

    2. WebSocket Ping/Pong Keepalive: Implement periodic ping frames to detect dead connections proactively, before the next request is sent.

    3. Connection Health Check: Before reusing a WebSocket connection, perform a lightweight health check.

    Code Changes

    1. Add first_event_timeout to Provider Configuration

    File: codex-rs/model-provider-info/src/lib.rs

    // Add new constant (after existing timeout constants)
    const DEFAULT_FIRST_EVENT_TIMEOUT_MS: u64 = 60_000;  // 60 seconds
    
    // In ModelProviderInfo struct, add field:
    pub struct ModelProviderInfo {
        // ... existing fields ...
        /// Timeout (in milliseconds) to wait for the first event after sending a
        /// WebSocket request. This should be shorter than stream_idle_timeout_ms
        /// to detect dead connections quickly.
        pub first_event_timeout_ms: Option<u64>,
        // ...
    }
    
    // Add method:
    impl ModelProviderInfo {
        // ... existing methods ...
        
        /// Effective timeout for receiving the first event after sending a request.
        pub fn first_event_timeout(&self) -> Duration {
            self.first_event_timeout_ms
                .map(Duration::from_millis)
                .unwrap_or(Duration::from_millis(DEFAULT_FIRST_EVENT_TIMEOUT_MS))
        }
    }
    
    // In to_api_provider():
    pub fn to_api_provider(&self, auth_mode: Option<AuthMode>) -> CodexResult<ApiProvider> {
        // ... existing code ...
        Ok(ApiProvider {
            // ... existing fields ...
            stream_idle_timeout: self.stream_idle_timeout(),
            first_event_timeout: self.first_event_timeout(),  // NEW
        })
    }
    
    // In create_openai_provider():
    pub fn create_openai_provider(base_url: Option<String>) -> ModelProviderInfo {
        ModelProviderInfo {
            // ... existing fields ...
            first_event_timeout_ms: None,  // NEW - uses default
            // ...
        }
    }

    2. Update Provider Struct

    File: codex-rs/codex-api/src/provider.rs

    #[derive(Debug, Clone)]
    pub struct Provider {
        pub name: String,
        pub base_url: String,
        pub query_params: Option<HashMap<String, String>>,
        pub headers: HeaderMap,
        pub retry: RetryConfig,
        pub stream_idle_timeout: Duration,
        pub first_event_timeout: Duration,  // NEW
    }

    3. Update WebSocket Connection to Use First-Event Timeout

    File: codex-rs/codex-api/src/endpoint/responses_websocket.rs

    pub struct ResponsesWebsocketConnection {
        stream: Arc<Mutex<Option<WsStream>>>,
        idle_timeout: Duration,
        first_event_timeout: Duration,  // NEW
        server_reasoning_included: bool,
        models_etag: Option<String>,
        server_model: Option<String>,
        telemetry: Option<Arc<dyn WebsocketTelemetry>>,
        last_activity: Arc<std::sync::Mutex<std::time::Instant>>,  // NEW for health check
    }
    
    impl ResponsesWebsocketConnection {
        fn new(
            stream: WsStream,
            idle_timeout: Duration,
            first_event_timeout: Duration,  // NEW parameter
            server_reasoning_included: bool,
            models_etag: Option<String>,
            server_model: Option<String>,
            telemetry: Option<Arc<dyn WebsocketTelemetry>>,
        ) -> Self {
            Self {
                stream: Arc::new(Mutex::new(Some(stream))),
                idle_timeout,
                first_event_timeout,  // NEW
                server_reasoning_included,
                models_etag,
                server_model,
                telemetry,
                last_activity: Arc::new(std::sync::Mutex::new(std::time::Instant::now())),  // NEW
            }
        }
    
        /// Check if the connection appears healthy based on last activity time.
        /// Returns false if the connection has been idle for longer than the
        /// first_event_timeout, suggesting it may be dead.
        pub async fn is_healthy(&self) -> bool {
            let last_activity = self.last_activity.lock().unwrap();
            last_activity.elapsed() < self.first_event_timeout * 2
        }
    
        // In stream_request():
        pub async fn stream_request(
            &self,
            request: ResponsesWsRequest,
            connection_reused: bool,
        ) -> Result<ResponseStream, ApiError> {
            // ... existing setup code ...
            
            tokio::spawn(async move {
                // ... existing header event emission ...
                
                let mut guard = stream.lock().await;
                let result = {
                    let Some(ws_stream) = guard.as_mut() else {
                        let _ = tx_event.send(Err(ApiError::Stream(
                            "websocket connection is closed".to_string(),
                        ))).await;
                        return;
                    };
    
                    run_websocket_response_stream(
                        ws_stream,
                        tx_event.clone(),
                        request_body,
                        idle_timeout,
                        first_event_timeout,  // NEW parameter
                        telemetry,
                        connection_reused,
                        last_activity,  // NEW parameter
                    ).await
                };
                // ... error handling ...
            });
            // ...
        }
    }

    4. Update run_websocket_response_stream with Separate Timeouts

    File: codex-rs/codex-api/src/endpoint/responses_websocket.rs

    async fn run_websocket_response_stream(
        ws_stream: &mut WsStream,
        tx_event: mpsc::Sender<Result<ResponseEvent, ApiError>>,
        request_body: Value,
        idle_timeout: Duration,
        first_event_timeout: Duration,  // NEW parameter
        telemetry: Option<Arc<dyn WebsocketTelemetry>>,
        connection_reused: bool,
        last_activity: Arc<std::sync::Mutex<std::time::Instant>>,  // NEW parameter
    ) -> Result<(), ApiError> {
        send_websocket_request(
            ws_stream,
            request_body,
            first_event_timeout,  // Use shorter timeout for send
            telemetry.as_ref(),
            connection_reused,
        ).await?;
    
        let mut first_event = true;
        
        loop {
            let poll_start = Instant::now();
            
            // Use shorter timeout for first event, longer for subsequent events
            let event_timeout = if first_event {
                first_event_timeout
            } else {
                idle_timeout
            };
            
            let response = tokio::time::timeout(event_timeout, ws_stream.next())
                .await
                .map_err(|_| {
                    if first_event {
                        ApiError::Stream("idle timeout waiting for first websocket event after request (connection may be dead)".into())
                    } else {
                        ApiError::Stream("idle timeout waiting for websocket".into())
                    }
                });
                
            if let Some(t) = telemetry.as_ref() {
                t.on_ws_event(&response, poll_start.elapsed());
            }
            
            let message = match response {
                Ok(Some(Ok(msg))) => {
                    // Update last activity timestamp
                    *last_activity.lock().unwrap() = std::time::Instant::now();
                    msg
                }
                Ok(Some(Err(err))) => {
                    return Err(ApiError::Stream(err.to_string()));
                }
                Ok(None) => {
                    return Err(ApiError::Stream(
                        "stream closed before response.completed".into(),
                    ));
                }
                Err(err) => {
                    return Err(err);
                }
            };
    
            first_event = false;  // Subsequent iterations use idle_timeout
    
            // ... rest of event processing (unchanged) ...
        }
    }

    5. Update WebSocket Client Connection to Pass First-Event Timeout

    File: codex-rs/codex-api/src/endpoint/responses_websocket.rs

    impl ResponsesWebsocketClient {
        pub async fn connect(
            &self,
            extra_headers: HeaderMap,
            default_headers: HeaderMap,
            turn_state: Option<Arc<OnceLock<String>>>,
            telemetry: Option<Arc<dyn WebsocketTelemetry>>,
        ) -> Result<ResponsesWebsocketConnection, ApiError> {
            // ... existing connection code ...
            
            Ok(ResponsesWebsocketConnection::new(
                stream,
                self.provider.stream_idle_timeout,
                self.provider.first_event_timeout,  // NEW
                server_reasoning_included,
                models_etag,
                server_model,
                telemetry,
            ))
        }
    }

    6. Add WebSocket Ping/Pong Keepalive (Optional but Recommended)

    File: codex-rs/codex-api/src/endpoint/responses_websocket.rs

    // In WsStream::new(), add keepalive task:
    impl WsStream {
        fn new(inner: WebSocketStream<MaybeTlsStream<TcpStream>>) -> Self {
            let (tx_command, mut rx_command) = mpsc::channel::<WsCommand>(32);
            let (tx_message, rx_message) = mpsc::unbounded_channel::<Result<Message, WsError>>();
    
            // NEW: Periodic ping keepalive
            let ping_tx = tx_command.clone();
            tokio::spawn(async move {
                let mut interval = tokio::time::interval(Duration::from_secs(30));
                interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
                loop {
                    interval.tick().await;
                    let (tx_result, mut rx_result) = oneshot::channel();
                    if ping_tx.send(WsCommand::Send {
                        message: Message::Ping(Vec::new()),
                        tx_result,
                    }).await.is_err() {
                        break;  // Channel closed, exit keepalive
                    }
                    // If ping fails, the connection is dead - signal through rx_message
                    if let Ok(Err(e)) = rx_result.await {
                        let _ = tx_message.send(Err(e));
                        break;
                    }
                }
            });
    
            let pump_task = tokio::spawn(async move {
                // ... existing pump task code (unchanged) ...
            });
    
            Self {
                tx_command,
                rx_message,
                pump_task,
            }
        }
    }

    7. Update Config Schema

    File: Update config schema to include new field

    The stream_idle_timeout_ms config option already exists. Add first_event_timeout_ms:

    # In ~/.codex/config.toml under [model_providers.openai]:
    stream_idle_timeout_ms = 300000      # Existing: 5 min idle timeout during streaming
    first_event_timeout_ms = 60000       # New: 1 min timeout for first event after request

    Testing Strategy

    Unit Tests

    1. First-Event Timeout Test:

      • Create a mock WebSocket that delays the first event by 90s
      • Verify the request times out after 60s (first_event_timeout), not 300s
      • Verify subsequent events use the full 300s idle timeout
    2. Dead Connection Detection Test:

      • Create a mock WebSocket that accepts send but never responds
      • Verify timeout fires after 60s
      • Verify error message distinguishes first-event timeout from idle timeout
    3. Keepalive Ping Test:

      • Verify ping frames are sent every 30s
      • Verify failed ping triggers connection failure
      • Verify pong responses reset the activity timer

    Integration Tests

    1. End-to-End Tool Execution Test:

      • Run a turn with multiple tool calls
      • Simulate connection drop between tool execution and next request
      • Verify reconnection happens within 60s, not 300s
    2. Windows-Specific Test:

      • Test on Windows with VS Code integrated terminal
      • Verify the fix resolves the 300s stall
      • Measure actual stall time before and after fix
    3. Long-Running Turn Test:

      • Run a turn with many tool calls over extended period
      • Verify connection stays healthy with keepalive
      • Verify idle timeout still works for genuine stalls

    Regression Tests

    1. Normal Operation Test:

      • Verify no change in behavior for healthy connections
      • Verify first event arrives well within 60s on normal connections
      • Verify 300s idle timeout still protects against genuine stream stalls
    2. HTTP Fallback Test:

      • Verify HTTP path continues to work correctly
      • Verify HTTP path doesn't use first_event_timeout (not applicable)

    Performance Tests

    1. Timeout Impact Measurement:

      • Measure time from tool result to next model request
      • Before fix: ~300s (when connection is dead)
      • After fix: ~60s (when connection is dead), ~instant (when healthy)
    2. Keepalive Overhead:

      • Measure network overhead of ping/pong frames
      • Expected: negligible (4 bytes per 30s)

    Impact Assessment

    Positive Impacts

    1. Reduced Stall Time: Dead connections detected in 60s instead of 300s (80% reduction)
    2. Better Error Messages: Distinguishes first-event timeout from idle timeout
    3. Proactive Detection: Keepalive ping/pong detects dead connections before they cause stalls
    4. Configurable: Users can tune first_event_timeout_ms based on their network conditions
    5. Backward Compatible: Default 60s first-event timeout is well within normal response times

    Risk Assessment

    1. Low Risk: 60s timeout is generous for first event (normal responses arrive in <10s)
    2. No Breaking Changes: Existing configurations continue to work; new field is optional
    3. Graceful Degradation: If keepalive fails, falls back to timeout-based detection
    4. HTTP Path Unaffected: HTTP path creates new connections per request, no change needed

    Edge Cases

    1. Slow Models: Some models may take 30-60s for complex reasoning. The 60s first-event timeout accommodates this.
    2. Rate Limiting: If rate-limited, the server responds immediately with 429, not a timeout.
    3. Server Overload: Server overloaded responses come as events, not timeouts.
    4. Very Long Tool Execution: Tool execution happens between model requests, not during streaming. This fix doesn't affect tool execution time.

    Configuration Recommendations

    For users experiencing connectivity issues:

    [model_providers.openai]
    # Reduce first-event timeout for faster dead connection detection
    first_event_timeout_ms = 30000  # 30 seconds
    
    # Keep idle timeout high for long streaming responses
    stream_idle_timeout_ms = 300000  # 5 minutes (default)

    For users on stable connections (no change needed):

    • Default values work well: 60s first-event, 300s idle

    Files Modified

    File Changes
    codex-rs/model-provider-info/src/lib.rs Add first_event_timeout_ms field, constant, and method
    codex-rs/codex-api/src/provider.rs Add first_event_timeout to Provider struct
    codex-rs/codex-api/src/endpoint/responses_websocket.rs Use separate timeouts, add keepalive, add health check
    codex-rs/codex-api/src/lib.rs Export new types if needed
    Config schema Add first_event_timeout_ms documentation

    Alternative Approaches Considered

    1. Reduce stream_idle_timeout Globally

    Rejected: Would cause false positives on long streaming responses (e.g., complex reasoning that produces events infrequently).

    2. TCP Keepalive Configuration

    Rejected: TCP keepalive is OS-level and affects all connections. Not configurable per-connection in reqwest/tungstenite easily. Also, TCP keepalive defaults are too long on Windows.

    3. Connection Pool with Health Checks

    Rejected: Adds significant complexity. The current single-connection-per-turn design is simpler and the first-event timeout approach achieves the same goal with less complexity.

    4. Always Create New WebSocket Connection

    Rejected: Would lose the benefits of connection reuse (sticky routing, previous_response_id caching, reduced latency). The keepalive + first-event timeout approach maintains connection reuse while detecting dead connections.


    Conclusion

    The 300-second stall is caused by the WebSocket connection reuse pattern combined with the 300-second default idle timeout. When the connection becomes stale or dead between model requests, the client waits the full 300 seconds before detecting the failure.

    The proposed fix introduces a separate 60-second timeout for the first event after sending a request, reducing the stall from 300s to 60s in the worst case. Combined with WebSocket ping/pong keepalive, dead connections can be detected proactively, often before the next request is even sent.

    This fix is backward compatible, low-risk, and addresses the root cause without changing the fundamental architecture of connection reuse.

  4. Avishayy commented on May 27, 2026

    @Avishayy

    We're hitting what looks like the same issue on codex-cli 0.133.0, and can add a couple of data points from packet + app-server traces that may help localize it.

    Same signature: periodic stalls where a turn hangs after dispatch with no output, then recovers cleanly. Our stalls that we let run to natural recovery come back at ~300s, matching the
    exact-300s timing reported here. Independent of context size, reasoning effort, or rate limits (rate limits were ~1% when it happened).

    Two differences/additions from this report:

    1. It also happens over the responses_websocket transport (ChatGPT-auth, not a custom HTTP endpoint), so it appears to be the responses wire API generally rather than HTTP-specific.
      The model client span shows the idle gap directly:

    model_client.stream_responses_websocket{model=gpt-5.5 wire_api=responses
    transport="responses_websocket" ...}: codex_core::client: close
    time.busy=3.74ms time.idle=304s

    i.e. the connection was open and idle for 304s with the turn awaiting frames, then closed.
    2. Packet capture shows at least one concrete trigger: a stale pooled connection that the server RSTs, which codex doesn't fast-fail. At the instant the turn dispatched, codex pushed
    the request onto a pooled connection the server had already closed and got an immediate RST:

    T+0.203  CLIENT.36670 > SERVER.443: Flags [P.]   # request pushed onto a pooled conn
    T+0.245  SERVER.443 > CLIENT.36670: Flags [R]    # server RST (~42ms later)
    #  other pooled conns then exchange only keepalive ACKs — no data, no retry —
    #  and no fresh connection is opened until the ~300s mark
    

    So it's not (in this case) backend silence or a dead path — the server actively reset a reused idle connection, and codex appears to wait the full ~300s idle window before surfacing it
    as stream disconnected - retrying instead of fast-failing on the RST. The retry itself works fine and recovers in <250ms once it fires — consistent with this report. So the painful
    part is the ~300s detection latency, not the retry.

    Questions for maintainers:

    • Is the ~300s governed by stream_idle_timeout_ms (it defaults to unset/None in ModelProviderInfo for the built-in provider)? If so, is there a recommended lower value, or should a
      pooled-connection RST trigger an immediate retry independent of the idle timeout?
    • Would surfacing a TCP RST on a pooled connection as a fast connection-error (→ retry on a fresh connection) be in scope, separate from the idle watchdog?

    Environment: codex-cli 0.133.0, app-server mode (driven via JSON-RPC), gpt-5.5, model_reasoning_effort=high, Linux. Happy to share a sanitized pcap excerpt or full codex_core::client /
    tungstenite frame trace if useful.

  5. rebroad commented on Aug 6, 2026

    @rebroad

    Adding one complementary point to the first-event-timeout/keepalive proposal above:

    The retry policy should also have a bounded aggregate budget, rather than allowing each of the 5 stream retries to consume the full 300-second idle timeout. Otherwise one stale connection can produce 25+ minutes of apparent “Working” (and reports show 30–45 minute cases).

    Possible design:

    • Keep separate connection/first-event and inter-event stream timeouts.
    • Add a per-turn stream-recovery deadline (for example, 2 minutes, configurable per provider).
    • Stop retrying when that deadline is exhausted and emit a terminal, user-visible error.
    • Show the retry number, elapsed time, next retry/deadline, and preserve an immediate cancel/interrupt path.
    • Reset the recovery budget only after actual stream progress, not merely after opening another connection.

    This should complement fast handling of TCP resets and ping/pong health checks: the transport can recover quickly when possible, but a repeatedly unhealthy path cannot leave the turn looking indefinitely active.

  6. HomCrazyEveryDay commented on Aug 12, 2026

    @HomCrazyEveryDay

    I had the same issue.

    Reconnecting 2/5
    stream disconnected before completion:
    idle timeout waiting for websocket

    300.003/300.021ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIbugSomething isn't workingconnectivityIssues involving networking or endpoint connectivity problems (disconnections)custom-modelIssues related to custom model providers (including local models)performancewindows-osIssues related to Codex on Windows systems

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions