response.incomplete is treated as a retryable stream failure, causing unnecessary retries and transport fallback #38831

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   connectivityIssues involving networking or endpoint connectivity problems (disconnections)Issues involving networking or endpoint connectivity problems (disconnections)

   on Aug 16, 2026
2. jdcodes1 commented on Aug 17, 2026

   

   jdcodes1

   on Aug 17, 2026

   More actions

   Confirmed on current main (1f41cc5) — the mapping and both downstream symptoms check out, and the surrounding code makes the fix shape fairly clear.

   The mapping. response.incomplete extracts only incomplete_details.reason and returns ApiError::Stream:

   codex/codex-rs/codex-api/src/sse/responses.rs

   Lines 453 to 463 in 1f41cc5

   	"response.incomplete" => {
   	let reason = event.response.as_ref().and_then(|response| {
   	response
   	.get("incomplete_details")
   	.and_then(|details| details.get("reason"))
   	.and_then(Value::as_str)
   	});
   	let reason = reason.unwrap_or("unknown");
   	let message = format!("Incomplete response returned, reason: {reason}");
   	return Err(ResponsesEventError::Api(ApiError::Stream(message)));
   	}

   Two things are notable in context:

   1. It's the least-differentiated classification in the file. The response.failed arm directly above it carefully fans out into fatal (MisalignmentPolicyViolation), non-retryable (InvalidRequest), ServerOverloaded, and Retryable { delay } — while response.incomplete, which is a successful terminal status of the Responses API (the server completed the stream and told you why it stopped), gets the same treatment as a dropped TCP connection.
   2. The usage block really is discarded. The arm never touches event.response.usage, unlike response.completed (L464-L481), which parses ResponseCompleted and forwards token_usage. So the tokens billed for the truncated response never reach session accounting — your stale-accounting observation is structural, not incidental.

   Why retries and the transport fallback follow. Once the error is a stream-class failure, the retry machinery has no way to distinguish it: handle_retryable_response_stream_error retries up to stream_max_retries and then, for any error it's handed, tries the WebSocket→HTTPS transport switch (core/src/responses_retry.rs#L86-L100 — the exact "Falling back from WebSockets to HTTPS transport. {err}" message you captured). Classification is the only gate; there is no semantic check at the fallback site. Retrying an identical request against max_output_tokens can only reproduce the condition, and against content_filter it re-submits flagged content.

   Fix shape that falls out of the existing structure:

   1. Parse the response object in the incomplete arm the same way response.completed does, and emit a terminal event (either ResponseEvent::Completed with an incompleteness marker, or a dedicated ResponseEvent::Incomplete { reason, token_usage }). That immediately restores usage accounting.
   2. Let the turn loop handle the reason semantically: max_output_tokens is the same condition the auto-compaction/token-limit machinery already exists for — surfacing "response truncated (max_output_tokens)" and/or triggering compaction is coherent; content_filter should be a non-retryable user-visible error like InvalidRequest.
   3. If a conservative intermediate step is preferred: classify incomplete as a non-retryable ApiError variant so the retry loop and transport fallback are skipped, even before the richer terminal-event handling lands. That alone eliminates the wasted retries/fallback and the misleading "stream disconnected" framing.

   Your mock-SSE repro would convert directly into the regression test for whichever variant lands: assert zero retries, no transport fallback, and non-stale usage after an incomplete event with a usage block.

   👍React with 👍1Reacted by Aji
3. danielchristiancazares

   added 9 commits that reference this issue on Sep 22, 2026

   fix(inference): preserve incomplete response usage without resampling

   ...

   ef01b11

   fix(inference): preserve incomplete response usage without resampling

   ...

   32ee387

   fix(inference): preserve incomplete response usage without resampling

   ...

   7df161f

   fix(inference): preserve incomplete response usage without resampling

   ...

   d5020df

   fix(inference): preserve incomplete response usage without resampling

   ...

   54a1c62

   fix(inference): preserve incomplete response usage without resampling

   ...

   98fa707

   fix(inference): preserve incomplete response usage without resampling

   ...

   d984419

   fix(inference): preserve incomplete response usage without resampling

   ...

   59a40bd

   fix(inference): preserve incomplete response usage without resampling

   ...

   c1c21c8
4. Water-Run commented on Oct 12, 2026

   

   Water-Run

   on Oct 12, 2026

   Last edited by Water-Run

   More actions

   Observed again on codex-cli 0.162.1 with gpt-6.1-sol / max, using ChatGPT authentication and the shared local app-server daemon on Fedora Linux 44 (x86_64). The user reports recurrent disconnects; the timeline below documents one captured occurrence on October 11–12, 2026.

   This adds a long-running goal case where an incomplete-response retry chain is followed by stale usage accounting and a terminal context-window error. I have not run the mock-SSE reproduction or verified the internal cause; the correlation is from persisted rollout events and local runtime logs.

   Configuration at the time of failure

   model = "gpt-6.1-sol"
   model_context_window = 872000
   model_auto_compact_token_limit = 600000
   model_auto_compact_token_limit_scope = "total"
   model_reasoning_effort = "max"

   The account's refreshed CLI model catalog advertises max_context_window=872000 and effective_context_window_percent=95; task events reported an effective window of 828400. This was the CLI-advertised maximum, rather than an arbitrary larger API limit. Automatic compaction had already succeeded twice in the same thread.

   Started with codex --yolo, with a persisted goal that repeatedly checked an independently supervised background job. An outer exec call was used to poll shell commands; multiple CLI clients shared the same daemon. HTTP/HTTPS/SOCKS proxy environment variables were present. A later doctor run intermittently reported WebSocket TLS handshake EOF while HTTPS reachability succeeded, so transport conditions are a possible additional factor.

   Timeline from persisted records (UTC)

   Time	Evidence
   Oct 11 15:00:54	Last actual tool call was issued.
   15:01:44	Last fresh usage report: input 404204, cached input 383360, output 771, total 404975; effective context window 828400.
   15:04:58	The separately supervised background job completed successfully, proven by its durable completion record and subsequent read-only artifact verification. The agent did not collect that completion.
   16:00, 17:00, 18:00, and later	Usage notifications continued to repeat the same last_token_usage values while new reasoning items were being persisted.
   19:06:11	Retry 4/5 reported Incomplete response returned, reason: max_output_tokens.
   19:13:24	Retry 5/5 reported a WebSocket close before response.completed.
   20:09:26	Runtime logged falling back to HTTP.
   20:09:49	Turn failed with codex_error_info=context_window_exceeded and no final assistant message.

   From completion of the last tool call to the terminal error, there were over five hours of reasoning/retry activity with no further tool calls. Immediately before failure, the last-request token counts still reflected the 15:01:44 report. The final token-count event replaced totals with zeros except total_tokens=828400, and the goal became blocked.

   Sanitized runtime excerpts:

   stream disconnected - retrying sampling request (4/5 ...)
   sampling_error=stream disconnected before completion: Incomplete response returned, reason: max_output_tokens
   
   stream disconnected - retrying sampling request (5/5 ...)
   sampling_error=stream disconnected before completion: websocket closed by server before response.completed
   
   falling back to HTTP
   
   Codex ran out of room in the model's context window. Start a new thread or clear earlier history before retrying.
   

   The user also saw the following CLI output, including zero displayed token usage for this thread:

   Disconnected from this task. Any running work continues.
   To reconnect, run:
     codex resume <thread-id>
   Token usage so far: total=0 input=0 output=0
   

   The CLI attachment message alone does not establish why that client detached, or whether inference was still running. The persisted turn independently proves it had failed with context_window_exceeded; the external job had finished roughly five hours earlier.

   Expected behavior / additional regression coverage

   The terminal max_output_tokens condition should remain distinct from retryable transport loss and retain usage accounting, as proposed in this issue. Please also cover the case of a large-context goal turn with already streamed reasoning: repeated incomplete responses or reconnects should not leave stale context estimates and a multi-hour retry chain ending in an unrecoverable context-window error. The CLI should make the failed/blocked turn visible on reattachment rather than relying only on a generic disconnection message.

   Local mitigation, not evidence of an upstream fix

   We preserved the CLI maximum window (872000) and the total-history compaction threshold (600000), set the recovered thread and an explicit long-task CLI profile to high, enabled model_post_turn_compact_threshold_percent=70, and disabled features.unbounded_connection_retries. Config loading and the daemon's effective config were verified. We also added instructions to use bounded tool-based polling and promptly audit background-job completion. These changes have not been validated by another multi-hour reproduction and do not fix the incomplete-response classification in Codex itself.

   The original rollout has been preserved locally. Full transcripts, private task details, host addresses, authentication material, and raw encrypted reasoning are intentionally excluded from this public report.
5. jif-oai commented on Oct 13, 2026

   

   jif-oai

   on Oct 13, 2026

   Contributor

   More actions

   fixing
6. jif-oai

   closed this as completedon Oct 13, 2026