Skip to content

Detect dead websocket connections faster after network changes instead of showing Working for up to 5 minutes #17003

Open
Open
Detect dead websocket connections faster after network changes instead of showing Working for up to 5 minutes#17003
Labels
bugSomething isn't workingconnectivityIssues involving networking or endpoint connectivity problems (disconnections)

Description

@0xdeafbeef
0xdeafbeef
opened on Apr 7, 2026

What version of Codex CLI is running?

codex-cli 0.118.0

What subscription do you have?

pro

Which model were you using?

No response

What platform is your computer?

it is platform independent

What terminal emulator and version are you using (if applicable)?

No response

What issue are you seeing?

When the network changes underneath an active Codex session, for example switching VPNs or after resuming from sleep, the TUI can keep showing Working for 5 minutes even though the underlying websocket is effectively dead.

The websocket response stream is considered dead only after stream_idle_timeout_ms, which defaults to 300000 ms (5 minutes).

Relevant code:

  • codex-rs/model-provider-info/src/lib.rs
    • DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000
    • DEFAULT_STREAM_MAX_RETRIES = 5
    • DEFAULT_WEBSOCKET_CONNECT_TIMEOUT_MS = 15_000
  • codex-rs/codex-api/src/endpoint/responses_websocket.rs
    • run_websocket_response_stream() waits with tokio::time::timeout(idle_timeout, ws_stream.next())
  • codex-rs/core/src/codex.rs
    • retry UI for websocket reconnects is partially hidden in release builds on the first retry

Because of that, a silent network break can look like a healthy in-progress turn until the 5-minute idle timeout expires.

What steps can reproduce the bug?

Switch vpn, turn wifi on/off/put laptop to sleep and resume.

What is the expected behavior?

Codex should detect dead websocket connections promptly after network changes and either:

  • reconnect within seconds, or
  • clearly show a reconnecting / waiting-for-network state instead of Working

Additional information

Any of these would be an improvement:

  • add proactive websocket heartbeat ping/pong from the client and fail the connection if pong is not received within a short deadline
  • lower the effective websocket idle detection timeout during active turns
  • surface the first websocket reconnect attempt in the TUI instead of hiding it in release builds
  • distinguish between “model is working” and “connection is stalled / reconnecting”

Activity

  1. 0xdeafbeef
    added
    bugSomething isn't working
    on Apr 7, 2026
  2. github-actions
    added
    CLIIssues related to the Codex CLI
    TUIIssues related to the terminal user interface: text input, menus and dialogs, and terminal display
    on Apr 7, 2026

  3. github-actions commented on Apr 7, 2026

    @github-actions
    github-actionsbot
    on Apr 7, 2026 – with GitHub Actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  4. etraut-openai
    added
    agentIssues related to the core agent loop
    and removed
    TUIIssues related to the terminal user interface: text input, menus and dialogs, and terminal display
    CLIIssues related to the Codex CLI
    on Apr 7, 2026
  5. alexstyl
    marked Stuck at working when changing network #17433 as a duplicate of this issue on Apr 11, 2026
  6. etraut-openai
    added
    connectivityIssues involving networking or endpoint connectivity problems (disconnections)
    and removed
    agentIssues related to the core agent loop
    on Apr 11, 2026
  7. henrymodisett
    added a commit that references this issue on Apr 27, 2026

    fix(codex): three ergonomic defaults from the OpenAI prompting resear…

    452fdd4
  8. DmitryPogodaev
    added a commit that references this issue on May 4, 2026

    fix(codex): plumb timedOut/idleTimedOut flags so embedded-runner retr…

    fb05aa4

  9. Ragnoroct commented on Jul 29, 2026

    @Ragnoroct
    Ragnoroct
    on Jul 29, 2026

    I traced my Linux suspend/resume stalls to IPv6 address rotation.

    After resume, NetworkManager has removed the WebSocket's source address, but
    Linux still reports the TCP socket as established (SO_ERROR=0, no EOF). Codex
    reuses it and waits for the 300-second timeout.

    Ragnoroct/codex@082a5fc is a diagnostic-only commit. It checks the socket's local IP before each send with a temporary UDP bind (no traffic):

    before suspend: assigned
after resume:   stale (EADDRNOTAVAIL)

  10. argothiel commented on Sep 13, 2026

    @argothiel
    argothiel
    on Sep 13, 2026

    Same issue after simple sleep/resume on codex-cli 0.154.0.

  11. mib00038 commented on Sep 27, 2026

    @mib00038
    mib00038
    on Sep 27, 2026

    Linux, codex-cli 0.157.1: same stall after suspend/resume. Tracing at 18344a9:

    • ResponsesWebsocket::is_closed() only checks whether the pump task has finished (codex-api/src/endpoint/responses_websocket.rs:225-231), so a half-open socket passes the reuse check in core/src/client.rs:1562-1571.
    • The client never sends pings, and incoming pongs are dropped (responses_websocket.rs:91-97).
    • connect_resolved_tcp sets only TCP_NODELAY (websocket-client/src/dialer.rs:209-217).
    • Detection falls to stream_idle_timeout (300 s) on tokio's monotonic clock, which stops during suspend, so the full 300 s runs after wake. Allow idle threads to prewarm and repair WebSocket connections #47701's idle repair also relies on is_closed().

    Proposed fix, two layers:

    1. Probe before reusing an idle socket (gives recovery in seconds).

    • Add WsCommand::Ping { tx_pong: oneshot::Sender<()> } to the pump. The pump sends Message::Ping and completes tx_pong on the next Message::Pong, instead of dropping it.
    • Add ResponsesWebsocketConnection::probe(deadline) -> bool, false if no pong arrives within about 2 s.
    • In the reuse check, when the connection has been idle more than about 15 s, measured on a clock that advances during suspend (CLOCK_BOOTTIME on Linux, SystemTime elsewhere), call probe. On false, reset with reason "probe_failed" and reconnect.
    • The first turn after wake then reconnects within about 2 s instead of 300 s, and turns on a recently used connection pay nothing.

    2. Kernel backstop in connect_resolved_tcp:

    let sock = socket2::SockRef::from(&stream);
sock.set_tcp_keepalive(&socket2::TcpKeepalive::new()
    .with_time(Duration::from_secs(30))
    .with_interval(Duration::from_secs(10)))?;
#[cfg(any(target_os = "linux", target_os = "android"))]
sock.set_tcp_user_timeout(Some(Duration::from_secs(20)))?;

    This bounds a send over a dead path to about 20 s and lets keepalive reap idle dead sockets. On its own it doesn't give seconds-level recovery, hence layer 1. socket2 0.6 is already a workspace dependency.

    We haven't tested a patch.

  12. Ragnoroct commented on Sep 27, 2026

    @Ragnoroct
    Ragnoroct
    on Sep 27, 2026

    On linux it can be a lot simpler.

    // Linux psuedocode: RTMGRP_IPV6_IFADDR + RTM_DELADDR
let ip = conn.local_addr()?.ip();

listen(RTMGRP_IPV6_IFADDR, |event| {
    if event.kind == RTM_DELADDR && event.ip == ip {
        reconnect(conn);
    }
});

  13. ninja- commented on Oct 4, 2026

    @ninja-
    ninja-
    on Oct 4, 2026

    I hope this gets fixed. my workflow involves re-connecting various devices posing as network adapters etc. even though I have a seperate one for stable internet connection. a single, irrelevant to internet access, netlink event moves the Codex websocket into a zombie state every single time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingconnectivityIssues involving networking or endpoint connectivity problems (disconnections)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions