Next sampling request is sent before the last tool output of a batch is appended, so the provider rejects it with "No tool output found for tool call X" and the thread stays wedged

What version of Codex App are you using?

Codex desktop app (app-server), cli_version 0.153.4 as recorded in the session's session_meta.

What platform is your computer?

Windows 11 (x64).

What issue are you seeing?

A thread becomes permanently unable to continue: every later user turn — and every fork of it — fails with

{"error":{"message":"No tool output found for tool call call_01_5kk0GRUtk7tY2lEwSKhY0823.","type":"invalid_request_error","param":null,"code":"invalid_request_error"}}

The trigger is a race inside a single assistant message that batches several tool calls. The follow-up
sampling request goes out before the last tool result of that batch has been appended to the input, so the
serialized request contains a function_call with no matching function_call_output. The provider rejects
the whole request, and the rejected turn is now part of the thread's live history, so the same 400 repeats
forever (a restart is the only thing that clears it).

I have the rollout JSONL for three wedged threads. The timelines below are the tool-call / tool-result
timestamps from those files plus the Turn error timestamp from the logs.

What steps can reproduce the bug?

Variant A — two view_image calls in one message (race window ≈ 70 ms). One assistant message issued two
image views; the two results were written back 68 ms apart, and the next request went out inside that gap:

17:10:33.270  function_call          view_image    call_00_si6t6wrnjTkCBhCbg89w4748
17:10:33.348  function_call          view_image    call_01_5kk0GRUtk7tY2lEwSKhY0823
17:10:34.115  item_completed         ImageView     call_00_si6t6wrnjTkCBhCbg89w4748
17:10:34.164  item_completed         ImageView     call_01_5kk0GRUtk7tY2lEwSKhY0823   <- batch now looks "done"
17:10:34.188  function_call_output   call_00_si6t6wrnjTkCBhCbg89w4748 (image)
17:10:34.189  message (developer)    <image_resize_notice>
17:10:34.256  function_call_output   call_01_5kk0GRUtk7tY2lEwSKhY0823 (image)       <- 68 ms too late
17:10:37.260  task_complete          error: No tool output found for tool call call_01_5kk0GRUtk7tY2lEwSKhY0823

The follow-up request therefore had to be issued between 34.189 and 34.256 — it was accepted by the transport
and rejected by the server ~3 s later.

Variant B — one view_image batched with a command that exceeded the 30 s tool wait. Here the race
window is ~30 s wide, so it reproduces every time:

16:53:36.098  function_call          view_image    call_00_xOnP62dcYl3n9S9KIgyj0114
16:53:36.981  function_call          exec_command  call_01_bAkMrxrT9wYGu6WvjHjs2192   (ran ~30 s)
16:53:37.052  function_call_output   call_00_xOnP62dcYl3n9S9KIgyj0114 (image)
16:53:37.052  message (developer)    <image_resize_notice>                       <- request goes out here
16:54:07.421  function_call_output   call_01_bAkMrxrT9wYGu6WvjHjs2192 (command result, 30 s late)
16:54:11.683  task_complete          error: No tool output found for tool call call_01_bAkMrxrT9wYGu6WvjHjs2192

Why this is a single-turn race and not a corrupt rollout

After the fact, the on-disk rollout for these threads is consistent — I scanned every response_item
in the affected session files (>8k items) and every *_call has a matching *_output by call_id. The
tool result is written a few tens of milliseconds after the request that needed it.
The wedge lives in the in-memory conversation: the unmatched call stays in the thread's live history,
which is why every later turn (and every fork) reproduces the identical 400 within ~1 s without hitting the
network. This matches Thread gets stuck after failed auto-compact; app-server keeps stale in-memory history until unload #31579 ("app-server keeps stale in-memory history until unload") — and the
archive → unarchive → notLoaded recovery noted in Inline image output costs ~230 tokens but 4.2 MB on the wire, so token-based context management never sees the payload that wedges the thread #41338 is consistent with that.
The app does not use provider-side continuation here: the captured request shows store=false and no
previous_response_id, i.e. the full history is re-sent every turn, so the unmatched call really is in the
body (it is just missing its output).

What is the expected behavior?

Gate the follow-up request on the whole batch. Do not issue the next sampling request until every tool
call from the current assistant message has its *_output item appended (the gap above is 68 ms; a
command that exceeds the tool wait limit leaves a 30 s window).
Re-validate pairing on the serialized input immediately before sending (and fail closed: if a call has
no output, either wait or synthesize one) instead of letting an unpaired call reach the provider.
Treat this 400 as recoverable. If the provider does reject it, re-send once with a stand-in output for
the named call_id rather than ending the turn — ending the turn is what makes the thread permanently
dead. (This is what my local workaround does, see below; the retry succeeds.)

Additional information

Provider: a third-party OpenAI-compatible endpoint (wire_api = "responses", DeepSeek). The provider
enforces "every tool call needs a tool output" strictly; I have not tested whether the OpenAI endpoint
accepts the same unpaired body.
Impact: 3 threads wedged in one evening of ordinary work (image-heavy review sessions), each with the same
error, and each unrecoverable in place. The work in those threads was only recoverable by redoing it in a
new thread.
Immediate workaround that works for us: keep view_image out of any batched message, and never batch a
command that can exceed the 30 s tool wait. Additionally, a small local proxy on the provider base_url
that (a) injects a stand-in output for an unpaired call and (b) retries once when the upstream returns this
400 has been verified to turn the failing request into a 200 (4/4 end-to-end cases, plus the original
10 unit tests for the unrelated duplicate-namespace repair it also carries).
Related: No tool output found for tool search call #31053 (same error via tool_search), exec_command never returns and yield_time_ms never fires: tool call orphaned with no output record #36827 (orphaned exec_command), Thread gets stuck after failed auto-compact; app-server keeps stale in-memory history until unload #31579, Inline image output costs ~230 tokens but 4.2 MB on the wire, so token-based context management never sees the payload that wedges the thread #41338, Desktop threads can be poisoned by inline base64 tool images, leading to {"detail":"Bad Request"} on resume and likely inflated token usage #18629.
Happy to provide the raw rollout files (or a trimmed excerpt) if that helps — say the word and I will
attach them with the local paths and identifiers redacted.

@mbj733

@mbj733

This race is exactly the sort of agent failure I’m trying to make recoverable. You already have a solid repro, if you’re open to it, I can take one failing tool batch and show you a Badgr run that waits for all tool outputs and retries from a clean state instead of permanently wedging the thread.

I reproduced this locally on macOS with Codex Desktop/app-server 0.154.0-alpha.6.2 and Cockpit Tools 1.3.52 using DeepSeek Responses.

Evidence from the persisted rollout:

A long Docker grader tool call call_00_ET_j75XEqb2hwcvyrTVc7MS0236 was issued at 18:56:39 local.
The following request returned HTTP 400 through the local Responses gateway at the same timestamp.
The rollout then recorded the tool result and immediately recorded No tool output found for tool call ....
Archive/unarchive changed the loaded thread to notLoaded, confirming stale app-server state; the persisted JSONL was not simply missing the output.

As an immediate provider-scoped mitigation, I prepared Cockpit PR #2431 (https://github.com/JunkaiWang-TheoPhy/cockpit-tools/pull/2431), which forces parallel_tool_calls=false only for api.deepseek.com Responses gateways. Focused tests and a patched sidecar smoke request pass.

The permanent fix still belongs in Codex app-server: gate the next sampling request on committed outputs for every call in the current batch, validate call/output pairing immediately before serialization, and reload/retry a rejected turn instead of permanently poisoning the thread.

Repair unpaired tool calls before DeepSeek Responses sees them

An update from the same setup (macOS, Codex Desktop/app-server 0.154.0-alpha.6.2, DeepSeek Responses through a local gateway), with a wire-level A/B that separates this race from the injected-standalone-output failure class.

Reproduction of this race (verified): sending a request whose input contains a function_call without its function_call_output makes DeepSeek answer 400 No tool output found for tool call .... The same body with the output appended answers 200. A separate shape — a standalone function_call_output with name/namespace but no call_id — answers 400 input: missing field 'call_id', and adding only a call_id is not a fix: with no matching function_call the upstream answers No tool call found for tool output with call_id ..., which matches the probes reported in #45227.

On the provider-side workaround: forcing parallel_tool_calls=false on the DeepSeek request does not stop batching. With the flag set, the same prompt still returned 3 function_call items in a single assistant turn on three consecutive runs. So a provider-side serialization toggle cannot be relied on to close this window; the request either needs a batch-commit barrier (your item 1) or a repair pass before it reaches the provider.

Interim mitigation implemented in the gateway (Cockpit Tools PR jlcodes99/cockpit-tools#2434, issue #2433): before forwarding, the gateway enforces "every tool call has a tool output" per request — a missing output is replaced by an explicit failure result (tool result unavailable..., never an empty success), and a standalone injected output gets the function_call it never had. Both failing bodies above then return 200, while complete requests are forwarded byte-identically. This is a per-request repair, not a fix for the in-memory poisoning: after a rejected turn the thread still needs unload/reload, which is why items 1–3 in your report remain the permanent fix on the app-server side.

Repair unpaired tool calls before strict Responses upstreams see them

Filed the missing-guard location separately as #46193 — it may be more actionable than another symptom report for this race.

Short version: normalize::ensure_call_outputs_present (which synthesizes an output for a dangling call) has exactly one call site, context_manager/history.rs:798, so it only runs when history is rebuilt; client.rs:849 build_responses_request does no pairing check; and the first sampling request of a turn uses the caller-provided initial_input (session/turn.rs:1585) rather than the normalized snapshot. So nothing prevents a request with a dangling call from leaving the process, which is consistent with your reproduction (the rollout is fine afterwards — the wire payload was not).

There is also a runnable strict-upstream stub in that issue that reproduces the provider's verdicts (including the misleading reasoning_text wording for an ordering problem), in case it helps to have a target to test a barrier against.

+1 with a timing data point that looks like exactly this race.

Setup: Windows 11, Codex desktop app 26.1007.21434 (codex-cli 0.162.0-alpha.17.2), custom strict provider (wire_api = "responses", OpenCode Zen / DeepSeek) behind a local OpenAI-compatible relay. One assistant step with two parallel tool calls:

20.921  function_call          view_image      call_00_hC0DonBjeKHawzABcyLL1642
21.127  function_call          exec_command    call_01_mNySew0rARWuQYQXPeIY2319
21.216  function_call_output   call_00_hC0...  (image, 1024x9000)
21.216  message (developer)    <image_resize_notice>
21.465  function_call_output   call_01_mNy...  (command output "PIL 11.3.0")
21.465  post-sampling token usage: model_needs_follow_up=true
21.481  zstd "Compressed request body ... pre_compression_bytes=424286"   <- follow-up body assembled
21.534  request leaves the machine
31.443  HTTP 400 back from the provider
31.444  Turn error: "No tool output found for tool call call_01_mNySew0rARWuQYQXPeIY2319."

What supports the race reading over "the rollout ordering is itself the bug":

the last tool output of the batch is appended 16 ms before the follow-up request body is assembled, which is inside the window this issue describes;
re-sending the items exactly as recorded in the rollout - same order, same 155 KB image, same interleaved <image_resize_notice> - against the same provider endpoint returns 200 (4/4), so the stored history is complete and it is the request that went out which was not.

Upstream request id for the failing request (their error includes it): 01271f1e-0095-4424-9a58-7ccb17b37182.

One caveat for anyone reproducing: whether a strict provider rejects the interleaved order can depend on the text of the interleaved item, not only on the order - measured at the HTTP level for this provider, <image_resize_notice>...</image_resize_notice> is accepted while any other text in the same position is rejected with this same error. Details in #52834 and filed provider-side as anomalyco/opencode#54698. So both classes are real, and a send-boundary pairing check plus fail-closed behaviour would cover both.

gateway: an item a batch's tool results were interrupted by goes afte…

fix: thinking by model list, account failover, cut streams, missing c…

Same wedge as this issue, plus one more trigger variant, reproduced on the latest desktop build.

Environment

Codex Desktop on Windows 11, app version 26.1023.50603 (prod channel, reported up-to-date at time of writing; also seen earlier on 26.1021.3705.0)
Custom OpenAI-compatible model provider (Kimi), Chat Completions wire protocol. The provider strictly validates that every tool_calls entry is followed by a matching tool message, so any orphan call is a hard 400.

What happened

In a long-running thread, the assistant issued a batched image-inspection step — the UI showed "Viewed 2 images" (two view_image calls in one assistant message).
At roughly the same moment the app's connection to the provider dropped; the UI showed "Reconnecting 1/5".
The image results rendered in the UI, but apparently were never persisted into the conversation state used for the next request — an orphan tool_call remained.
Every subsequent user turn in that thread fails with HTTP 400:

Invalid request: an assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: view_image:23

(The failing id view_image:23 is the provider's native <tool_name>:<index> tool-call id format.)

Impact

Identical to the main report: the thread is permanently wedged — every retry resends the same malformed history and gets the same 400. Forking the thread copies the poisoned history and fails the same way; only a brand-new thread recovers. Note this 400 is session-level: it never appears in the desktop logs under AppData\Local\Codex\Logs, which makes it hard to correlate after the fact.

Why this variant may be worth adding

The variants documented above are intra-batch races (the ~70 ms gap; the >30 s tool wait). This report shows the same persistence gap is also hit when a reconnect lands inside the batched view_image window — i.e., any transport hiccup during such a turn can wedge the thread. On proxied/metered connections that is not a rare event, and view_image results are the largest payloads of any built-in tool, which widens the vulnerable window compared to ordinary tool calls.

Fix ask

Same as #46193: enforce call/output pairing at the send boundary (synthesize an aborted output for dangling calls), not only when history is rebuilt. Until then, the user-side mitigations that work for us: one view_image per turn, pre-shrinking large images before viewing, and never sending a new message while the reconnect indicator is up.

One more data point on which workloads this hits hardest, plus a mitigation pattern that has worked well for us since the report above.

Worst-case workload: iterative visual-inspection loops

The triggering workflow for us was document/diagram automation (Feishu/Lark whiteboards): the agent draws a flowchart, exports a preview PNG, view_images it to check the layout, adjusts, and repeats. Every visual check is an image-heavy turn, so a single reconnect anywhere in the loop poisons the whole thread — which by then is long and context-heavy. Iterative "render → look → adjust" agent loops are arguably the worst-case workload for this bug: not an occasional image view, but dozens of dice rolls per session.

Mitigation pattern: validate the structure, not the pixels

We put the discipline into the global AGENTS.md so it applies automatically across sessions:

- view_image: one image per message, never batched with other tools;
  pre-shrink anything >1280px before viewing (small Python helper).
- For artifacts that have a structured representation, validate textually
  instead of visually: SVG export / raw node JSON for whiteboard diagrams,
  DOM or source for web pages. Reserve view_image for one final acceptance
  pass on the shrunken image.
- Never send messages or new tool calls while the reconnect indicator is up.
- A poisoned thread is dead: new thread, never fork; checkpoint progress to
  files so work can resume.

The second rule turned out to be the biggest win by far: most "does the diagram look right" checks can be answered from the SVG / node JSON (positions, overlaps, edges, labels) with zero image payload. In a typical flowchart session that cuts view_image calls from ~dozens down to 1–2 — and the remaining ones are small. It doesn't fix the underlying race, but it shrinks the attack surface to near zero for this class of work.

Navigation Menu

Repository navigation

Next sampling request is sent before the last tool output of a batch is appended, so the provider rejects it with "No tool output found for tool call X" and the thread stays wedged #44604

What version of Codex App are you using?

What platform is your computer?

What issue are you seeing?

What steps can reproduce the bug?

Why this is a single-turn race and not a corrupt rollout

What is the expected behavior?

Additional information

Environment

What happened

Impact

Why this variant may be worth adding

Fix ask

Worst-case workload: iterative visual-inspection loops

Mitigation pattern: validate the structure, not the pixels

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Next sampling request is sent before the last tool output of a batch is appended, so the provider rejects it with "No tool output found for tool call X" and the thread stays wedged #44604

Description

What version of Codex App are you using?

What platform is your computer?

What issue are you seeing?

What steps can reproduce the bug?

Why this is a single-turn race and not a corrupt rollout

What is the expected behavior?

Additional information

Activity

michaelmanly commented on Sep 11, 2026

JunkaiWang-TheoPhy commented on Sep 14, 2026

JunkaiWang-TheoPhy commented on Sep 14, 2026

JunkaiWang-TheoPhy commented on Sep 17, 2026

Edward4226 commented on Oct 12, 2026

Lizardon386 commented on Oct 28, 2026

Environment

What happened

Impact

Why this variant may be worth adding

Fix ask

Lizardon386 commented on Oct 28, 2026

Worst-case workload: iterative visual-inspection loops

Mitigation pattern: validate the structure, not the pixels

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions