WebSocket fallback should recover after transient transport failures #23665

Activity

1. github-actions

   added

   enhancementNew feature or requestNew feature or request

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   connectivityIssues involving networking or endpoint connectivity problems (disconnections)Issues involving networking or endpoint connectivity problems (disconnections)

   on May 20, 2026
2. Tonyfenwick1982 commented on Sep 5, 2026

   

   Tonyfenwick1982

   on Sep 5, 2026 – with ChatGPT Codex Connector

   More actions

   Follow-up with measured bandwidth impact on Windows, and a request for bounded, payload-aware WebSocket recovery. The affected user explicitly authorized this diagnostic report.

   Environment: Windows 11 x64, Codex desktop 26.901.5003.0, selected app-server codex-cli 0.153.3, ChatGPT authentication, an explicit local HTTP proxy. HTTP_PROXY and HTTPS_PROXY were set, and the desktop was fully restarted.

   The explicit proxy workaround reported in #29958 successfully restored Responses WebSocket connections. Several subsequent coding tasks established successful WebSockets over multiple hours. It did not eliminate later image-heavy send failures.

   Sanitized observation on 2026-09-05 (UTC):

   • 08:25:57 through 08:26:25: an image-heavy task exhausted five retries with "failed to send websocket request" and Windows socket error 10054. New WebSocket handshakes succeeded during the retry sequence, but the request sends still failed.
   • 08:26:34: the client logged "falling back to HTTP".
   • 08:26:34 through 08:34:07: retained compression logs show 12 HTTP request bodies totaling 769,949,302 compressed bytes, with a maximum of 66,458,221 bytes per request.
   • One compacted local history record was 92,134,705 bytes. Its structural inspection found 43 inline image data URLs totaling 91,840,017 ASCII characters. This is consistent with the failure family in Responses WebSocket falls back when compacted replacement_history contains large inline images #24550.
   • A separate 90.25-second physical-interface sample recorded 71,602,589 outbound bytes during that activity. This is a whole-machine counter, not per-process accounting.

   Request-body preparation does not prove full delivery, and rotating logs are incomplete. The proxy also logged upstream resets, but its available entries lack timestamps; they do not establish which component caused the WebSocket resets. No decryption or large-payload A/B replay was performed.

   Requested recovery behavior:

   1. After a transient failure, use HTTP temporarily. After roughly 60-120 seconds, let the next eligible model request make one WebSocket recovery attempt. Do not interrupt an active response or generate background inference solely to probe connectivity.
   2. If recovery fails, increase the cooldown with jitter and cap the number and byte cost of recovery attempts. A successful handshake alone must not reset that budget; require a completed real response.
   3. At a transport transition, rebuild coherent conversation state. Reuse previous_response_id only when it is actually valid for the connection/provider. Do not duplicate already-executed tools or blindly replay an ambiguously accepted request.
   4. Make retries payload-aware. Repeated send failures on the same oversized history should not cause the same tens-of-MB image payload to be retried every minute. Diagnose explicit size errors separately from timeouts/resets; a reset alone does not prove a size limit.
   5. Add a per-task repeated-upload budget and a visible pause/recovery path. Use actual byte accounting where possible; clearly label prepared-body estimates when those are the only available measurements.
   6. Address historical inline images independently: preserve original assets and decision-critical visual information, prefer supported reusable references or a compact visual summary with on-demand reload, and avoid keeping unnecessary full-resolution copies in every compacted history. Never silently discard required image evidence.

   The user-facing goal is that an ordinary network interruption recovers automatically and an expensive repeated-upload loop stops with one actionable explanation. Users should not have to inspect transport logs, manually restart the app, and rebuild tasks repeatedly.

   Only aggregate operational measurements and public issue references are shared here. No raw sessions, prompts, project names, local paths, image contents, account identifiers, credentials, or private endpoints are attached.
3. Vantrongs commented on Sep 18, 2026

   

   Vantrongs

   on Sep 18, 2026

   More actions

   Additional measured case: successful auto-compaction can make the request small enough for WebSockets again, but the running client remains on HTTP until restart. This provides a concrete recovery trigger for this issue, beyond a generic cooldown. Related size/retry report: #38846; image-history report: #43015.

   Environment and limits of this report

   Observed September 18, 2026: CLI 0.155.0-alpha.9.2 as app-server in a locally patched Linux Desktop 26.915.31029, NixOS, ChatGPT authentication, gpt-6-astra, built-in OpenAI Responses endpoint. This is not an untouched Desktop installation. No transport/retry modifications were installed during this diagnosis, and no explicit HTTP/HTTPS/ALL proxy environment variables were present in the relevant processes.

   Runtime evidence is from that version. I also inspected the same session-latched fallback in current stable rust-v0.155.0 and latest prerelease rust-v0.156.0-alpha.2; I have not reproduced the runtime sequence on those two releases.

   Observed sequence (UTC)

   Time	Observation
   07:52–07:53	Long screenshot-heavy computer-use session: five stream retries, then HTTP fallback. HTTP body measured 23,784,376 bytes before compression, 16,880,511 after zstd; HTTP returned 200 and work continued without restart.
   08:09:34	After a full app restart, the client established WebSocket transport again.
   08:10:21–08:10:33	Auto-compaction hit the same generic WS close error, retried twice, then switched to HTTP. Its HTTP request was 24,173,203 bytes before compression, 17,026,309 after zstd.
   08:13:55	Completed compaction was recorded in the rollout.
   08:13:55–08:14:55	Subsequent real requests remained on responses_http, despite bodies now being only 9,339,825–9,371,272 bytes before compression (roughly 6.75 MB after zstd). HTTP responses succeeded.
   08:15:22	After another full restart, WebSocket transport connected again.
   08:16:30 and 08:16:52	The same task continued through real model/tool exchanges on responses_websocket, with auth_connection_reused=true.

   The error shown to the user was:

   stream disconnected before completion: websocket closed by server before response.completed
   

   These are prepared HTTP-body sizes, not measured network traffic or exact WS serialization sizes. The original incident's close code was not preserved by the client, so I cannot claim that its actual close frame was recorded as 1009.

   Separately, a bounded synthetic transport probe against the same official endpoint used a minimal valid response.create with generate:false, padded only with JSON whitespace (no user conversation or screenshots). Results:

   Text-message size	Result
   16,777,215 bytes (16 MiB minus 1)	response.completed
   16,777,217 bytes (16 MiB plus 1)	peer Close 1009, empty reason
   4,096 bytes immediately afterward	response.completed

   permessage-deflate was negotiated. This measured boundary is therefore on the decompressed message, not solved simply by enabling WS compression. It is an observed endpoint limit, not a claim about a documented permanent API contract.

   Code and requested recovery behavior

   In current prerelease ModelClient::force_http_fallback, disable_websockets.swap(true, ...) latches fallback. responses_websocket_enabled then rejects WS for this client. I found no reset of this flag after compaction; a new client initializes it to false.

   If the WS message-size policy cannot be increased, please consider one bounded, payload-aware recovery attempt on the next real request after successful compaction, when the complete serialized WS request is eligible again. Preserve valid conversation state; do not reuse a stale previous_response_id or replay already-executed tools. Require a completed response, not merely a successful handshake, to count recovery as successful. If recovery fails, retain working HTTP with a bounded retry/cooldown policy rather than entering another retry loop. Compaction alone is not proof of eligibility: retained user images can still leave the full request oversized.

   The initial oversized-request classification belongs to #38846: preserve Close code/reason and avoid repeating an unchanged request after explicit 1009. The additional point here is that a size-induced fallback need not remain permanent after the payload has genuinely shrunk.

   Long computer-use sessions repeatedly add screenshots, so this is a plausible recurring workload, not just a large text prompt. Restoring eligible WS continuation could avoid repeated full-history uploads and reduce latency/bandwidth. This report does not establish a controlled performance multiplier or reduced token billing.

   Only sanitized aggregate measurements are included; no raw sessions, image contents, account identifiers, credentials, or private project data are attached.