Long CLI threads exceed Responses WebSocket message limit before auto-compaction (close code 1009) #38846

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   connectivityIssues involving networking or endpoint connectivity problems (disconnections)Issues involving networking or endpoint connectivity problems (disconnections)

   contextIssues related to context management (including compaction)Issues related to context management (including compaction)

   on Aug 16, 2026
2. jdcodes1 commented on Aug 17, 2026

   

   jdcodes1

   on Aug 17, 2026

   More actions

   Excellent forensics — the patched-build close-code capture pins this. Confirming the code side on main @ 1f41cc5 and adding the classification/fix seams.

   The close frame really is discarded. The Responses WebSocket receive loop pattern-matches the close frame away and emits a fixed string:

   codex/codex-rs/codex-api/src/endpoint/responses_websocket.rs

   Lines 817 to 821 in 1f41cc5

   	Message::Close(_) => {
   	return Err(ApiError::Stream(
   	"websocket closed by server before response.completed".into(),
   	));
   	}

   Message::Close(_) → ApiError::Stream("websocket closed by server before response.completed"). Your diagnostic patch (including code + reason) is essentially the first fix as-is — without it, 1009 is indistinguishable from a flaky network drop, which is why the stock CLI's behavior looks so irrational.

   Why five identical retries follow. The request is serialized and sent as a single Message::Text (responses_websocket.rs#L882, #L904). Once the failure is an ApiError::Stream, the retry machinery treats it like any dropped stream: stream_max_retries same-transport retries, then the transport fallback (core/src/responses_retry.rs#L86-L100 — the gate has no semantic check on the error). For 1009 the payload is byte-for-byte identical on every retry, so all five attempts are deterministic failures; only the HTTPS fallback (no equivalent per-message cap) can succeed. This is the same classification gap as #38831 (response.incomplete → generic Stream error → blind retries + fallback): semantically terminal information is flattened into "stream broke".

   Why compaction never saves you. Auto-compaction triggers on token thresholds (your observed 244,800 of a 258,400 window), but the WS peer's limit is a byte cap on the serialized message. Tokens and bytes correlate loosely at best — histories heavy in JSON structure, tool output, or base64 content hit the byte cap well before the token threshold, opening exactly the window you measured (~225k tokens, under the compaction threshold, over the message cap). No token-tuned threshold can close a byte-defined gap.

   Fix outline, smallest first:

   1. Include close code + reason in the error (your patch). Also worth mapping WsError::ConnectionClosed separately from a received close frame — today both collapse into unlabeled Stream errors (#L567-L569).
   2. Classify a received close 1009 as deterministic-for-this-payload: skip same-transport retries and go straight to HTTPS fallback. That alone removes the five wasted attempts and most of the latency you're seeing.
   3. Byte-aware pre-flight: the client knows the serialized request size at #L904 before sending. Comparing it against a configured/known WS message cap and preferring HTTPS (or triggering early compaction) up front turns the failure mode into a non-event, and gives auto-compaction a byte-based trigger to complement the token one.

   Your reproduction (small prompt on a ~225k-token thread) is directly reusable as the regression test for (2)/(3): assert one WS attempt max and no user-visible retry storm when the serialized request exceeds the cap.
3. wegfawefgawefg commented on Aug 17, 2026

   

   wegfawefgawefg

   on Aug 17, 2026

   Author

   More actions

   when i instrumented it with heavier logging, it just said it was too big 5 times in a row.
   The solution is pretty simple. the client just needs to truncate the message under the cap when it sends it in that's all.
   I do not think the right behaviour is to skip retries or instantly default to http requests when the message goes above a certain size.
   Hell, subdividing the message would be fine, but if the cap is about throughput anyways, truncation seems right to me.

   That being said that call is for OpenAI to make, not me.

   I had codex chase a little local patch, but its trivial and openai should probably make one go into mainstream codex-cli on next version.
   I would guess a lot of users ran into this same issue and may not even realize it.
4. jdcodes1 commented on Aug 17, 2026

   

   jdcodes1

   on Aug 17, 2026

   More actions

   Fair pushback on the retry-skipping point — one consideration on truncation, though: the WS message here isn't a stream of independent data, it's one serialized JSON request containing the conversation history. Truncating it under the cap byte-wise would produce invalid JSON (or, if done structurally, silently drop history items the model was supposed to see — changing model behavior with no user-visible signal). The system already has a semantically-safe mechanism for "history too big": compaction, which summarizes rather than drops, and is model-visible. That's why I'd frame the byte-triggered path as "compact early (or choose the transport without the cap)" rather than literal truncation — same trigger you're proposing, but routed through the mechanism that preserves request validity. Subdivision would need server-side reassembly semantics the Responses WS protocol doesn't currently expose. Agreed the final call is OpenAI's; either way the first step is the same one your patch demonstrated — stop discarding the close code.
5. zsw1234 commented on Oct 3, 2026

   

   zsw1234

   on Oct 3, 2026

   More actions

   Additional macOS reproduction: successful upgrade, then close 1009 for an image-heavy request

   Reproduced on 2026-10-03 with the desktop-bundled codex-cli 0.159.2, GPT-6.1 Sol, and ChatGPT authentication. All controls used the same existing local HTTP proxy route; no node changes or direct-connect comparison were used.

   Original desktop symptom: an existing image-heavy conversation established WebSocket connections successfully, retried five times with websocket closed by server before response.completed, and then completed through HTTPS fallback. The original logs did not preserve the close code.

   Controls and targeted reproduction:

   • Fresh, short CLI inference requests completed over WebSocket without retries or fallback, both with isolated configuration and with the existing user configuration.
   • A temporary diagnostic client used the same authenticated Responses endpoint and proxy, with generate: false and store: false. It recorded only event types, sizes, timings, and close-frame metadata.
   • Small prewarm: 273 serialized UTF-8 bytes, HTTP 101 upgrade, then response.completed with zero output items, in 1,276 ms.
   • Image-heavy prewarm: 30 existing input images, 19,969,785 serialized UTF-8 bytes before WebSocket compression, HTTP 101 upgrade, then a peer close with code 1009 and an empty reason, in 5,959 ms. No application response events preceded the close.

   The large prewarm reused the images present in the active history before the original failure, but it did not replay the complete original desktop request. This confirms a size-related failure on that endpoint/route, not the exact historical close code or that every reconnect has the same cause. The precise server limit and the component enforcing it were not measured. The diagnostic client offered permessage-deflate; the negotiated extension and compressed wire size were not recorded.

   RFC 6455 defines 1009 as a message too large for the receiving endpoint to process. On current main, the Responses receive loop still discards the close frame as Message::Close(_) and returns a generic stream error:

   https://github.com/openai/codex/blob/main/codex-rs/codex-api/src/endpoint/responses_websocket.rs

   Suggested narrow fix

   1. Preserve the close code as structured transport error metadata rather than flattening it into a generic disconnect. Keep diagnostics sanitized.
   2. On a received 1009, switch promptly to the existing HTTPS fallback instead of retrying the unchanged message five times. Preserve ordinary transient-disconnect retry behavior.
   3. Consider a serialized-byte preflight only against an explicitly supported/configurable budget, not a limit inferred from this one sample. Do not silently truncate conversation content or images.

   Regression coverage could use a mock peer closing with 1009: assert no same-payload WebSocket retry storm, successful HTTP fallback, and unchanged input content. Keep separate coverage for ordinary transient disconnects and unavailable fallback.

   This comment contains aggregate diagnostic results only. No private prompts, image data, raw session files, account identifiers, credentials, personal paths, or proxy-provider/node details are attached.
6. TheOne2006 commented on Oct 12, 2026

   

   TheOne2006

   on Oct 12, 2026 – with ChatGPT Codex Connector

   More actions

   Additional macOS reproduction: boundary bracket around 16 MiB, peer close 1009, then successful HTTP fallback

   Tested on 2026-10-13 with the desktop-bundled codex-cli 0.162.0-alpha.17.2, macOS, and ChatGPT authentication.

   The additional evidence is a controlled size-boundary test on wss://chatgpt.com/backend-api/codex/responses, independent of real conversation contents and model inference.

   Controlled reproduction

   Each trial opened a fresh authenticated WebSocket through the same existing local HTTP CONNECT proxy, with normal TLS certificate verification enabled. No proxy settings or nodes were changed between trials. Every handshake succeeded.

   The first message was deliberately invalid at the application layer:

   payload = json.dumps(
       {"type": "diagnostic.invalid_event", "padding": "a" * n},
       separators=(",", ":"),
   ).encode("utf-8")

   It was sent as one unfragmented text message. This lets the endpoint parse/reject small messages without requesting model generation. Only synthetic ASCII padding was sent; no conversation history, images, tool outputs, or private project data were replayed.

   Serialized UTF-8 message bytes	Result	Total elapsed from starting connection
   1,072	Application error, status 400	0.878 s
   1,048,624	Application error, status 400	1.311 s
   8,388,656	Application error, status 400	9.416 s
   16,777,136	Application error, status 400	17.265 s
   16,778,288	Peer Close 1009, empty reason	13.667 s
   17,825,840	Peer Close 1009, empty reason	16.841 s

   The application error was Expected a 'response.create' message as the first websocket event. It is the expected control result, not a successful inference.

   16 MiB = 16,777,216 bytes: the last application-parsed control is 80 bytes below that value, and the first rejected message is 1,072 bytes above it. This brackets the observed boundary around 16 MiB; it is not a claim to have measured the exact maximum accepted byte. Each listed size was tested once. Timings include connection setup and upload, rather than measuring a server timeout. A separate trial that collected only a Ping control frame is excluded from this table.

   Corresponding production observations

   Two existing image-heavy conversations showed the stock error and repeated fresh WebSocket handshakes, followed by successful HTTP fallback through the same proxy:

   • Case A: HTTP fallback instrumentation reported 16,815,366 bytes before zstd, 9,723,865 bytes after zstd. Fresh WebSocket attempts closed roughly 10–12 seconds after successful connection. After five retry notifications and the final WebSocket attempt, HTTP fallback resumed model output.
   • Case B: 34,219,091 bytes before zstd, 24,436,120 bytes after zstd. Fresh retry connections closed roughly 16–23 seconds after connection. HTTP fallback subsequently resumed model output. Before this failed request, the client logged incremental request failed, websocket reuse properties didn't match.

   In Case B, the previous turn had completed only about 25 seconds before the new request began; prolonged idle time is not required for this failure mode.

   The stock production logs omit close-frame metadata, so I cannot prove that every historical close was 1009 or that the HTTP body length exactly equals each WebSocket message length. The synthetic test directly establishes the size-related 1009 behavior on the same endpoint/route; the production sizes, repeatable timing, and working HTTP fallback are consistent with it.

   Client recovery gap

   The public rust-v0.162.0-alpha.17 source still discards the frame in Message::Close(_) and returns the generic websocket closed by server before response.completed stream error:

   https://github.com/openai/codex/blob/rust-v0.162.0-alpha.17/codex-rs/codex-api/src/endpoint/responses_websocket.rs#L827-L831

   This is the public baseline for the installed .17.2 desktop build, not an assertion that the two builds are byte-identical.

   Please preserve close code/reason, classify 1009 as non-retryable for an unchanged payload, and immediately use HTTP fallback. A preflight check of the actual serialized WebSocket message bytes could avoid the initial rejection as well. This should complement token-based compaction and preserve useful image/history content, rather than silently truncating it.

   No raw logs, rollout files, authentication headers/tokens, account identifiers, local paths, or conversation contents are included in this report.
7. mirsella commented on Oct 15, 2026

   

   mirsella

   on Oct 15, 2026

   More actions

   I’ve hit a closely matching failure in two image-heavy coding threads.

   Environment:

   • Linux, Codex CLI 0.162.1 via T3 Code’s codex app-server
   • ChatGPT authentication
   • GPT-6.1 Sol; another affected thread also failed on GPT-6 Astra
   • Sleev gateway 1.8.8 between Codex and the upstream Responses endpoint

   Both threads stopped during context compaction with:

   stream disconnected before completion: websocket closed by server before response.completed
   

   Sending continue immediately retriggered compaction and the same failure. Forking one affected conversation only helped temporarily.

   For one failed attempt, Sleev logged:

   Incoming request:      83,668,853 bytes
   Outgoing request:      17,900,648 bytes
   Retained image/files:  10
   Image/file bytes:      17,184,200
   

   The upstream connection was to wss://chatgpt.com/backend-api/codex/responses and closed after roughly five seconds. The outgoing payload exceeds the approximately 16 MiB boundary reported above. Our logs did not expose the close code for this request, so I can’t confirm it was 1009.

   Manual recovery using HTTP/SSE restored progress. Recovery also involved rollback/context handoff, so this wasn’t a controlled comparison of the exact same payload. The failed WebSocket attempts ended in failed turns rather than successful automatic HTTP recovery.

   Preserving the upstream close code/reason and falling back to HTTP for oversized requests would make this much easier to diagnose and recover from.