Skip to content

Code-mode exec silently degrades a long-running command into a full-context model polling loop (34.6M tokens burned after the task already completed) #38495

Open

Description

@blue199288

Summary

In code mode, when a script passed to exec outlasts the yield deadline, the CLI parks it as a numbered cell and hands control back to the model. There is no completion push and no timer — the only way to learn that the command finished is for the model to spend a turn calling wait(cell_id). Each such turn resubmits the entire conversation context.

The result is a busy-wait loop where "ask whether the process is done yet" costs exactly the same as "perform a full code analysis". In the session below, a single git worktree remove --force over a large directory tree (~9.5 minutes of real unlink time) produced 90 model turns, 21,610,789 input tokens and 3,837 output tokens — an input:output ratio of 5632:1 — and all of it happened after the agent had already delivered its result and written its report to disk.

This is the mechanism-level report behind the symptoms in #38335, #34115 and #38437.

Mechanism

  1. Model emits tools.exec_command({cmd, yield_time_ms, max_output_tokens}).
  2. Script exceeds the yield deadline (default 10000 ms). The CLI returns Script running with cell ID <n> — not an error, not a handle the runtime will follow up on.
  3. Resumption is pull, not push. The model must emit wait({cell_id, yield_time_ms, max_tokens}) to learn anything.
  4. Every wait is a full model turn: complete context re-sent, ~240k input tokens in, ~31 tokens out.
  5. The loop is not idle. There is no schedule_wakeup and no timer event in the rollout — it is a continuous turn chain that also holds a concurrency slot for its whole duration.

Two consequences worth stating explicitly:

  • exit_code cannot rescue this. Per the CLI's own contract, exit_code is present only "when the command finished during this call" — so while the process is genuinely still running, it is legitimately absent on every single poll. The model is being told the truth; the truth just costs 240k tokens to obtain.
  • git worktree remove (like rm -rf) prints nothing while unlinking. "Deleting file 80,000 of 134,000" and "hung forever" are byte-identical from the model's point of view, so it has no basis on which to stop polling.

Observed incident

Session 019ffc35-8a14-76b3-bd0b-7e6bfe18b62c, plan prolite, 2026-08-14 17:39:56Z → 18:15:09Z.
Total: 275 turns / 54,701,161 tokens.

Timeline:

turn/record event
638 task result written back (ok: true) — work complete
642 report artifact written to disk — deliverable complete
648 exec_command with a 7-statement cleanup compound, yield_time_ms: 30000
649 Script running with cell ID 88 / Wall time 11.0 seconds
651–663 wait{cell_id:"88", yield_time_ms:1000} × 5
664 Script completed
666 write_stdin({session_id: 1095, chars: "", yield_time_ms: 30000, max_output_tokens: 5000})
667 Script running with cell ID 89 / Wall time 11.0 seconds
668–931 16 byte-identical write_stdin execs + 67 wait calls, cells 89–98 and 103–107, every one at yield_time_ms: 1000, all returning net-empty output
932 model gives up and sends � (Ctrl-C) — 9.5 minutes later
935–938 task_complete

Note record 666: an inner yield_time_ms: 30000 nested inside a call whose own outer yield deadline is 10000 ms. The park is structurally guaranteed, every time.

The actual slow statement was the second of seven:

git -C repos/<repo> worktree remove ../../agent_workspace/<id>/worktrees/<name> --force

Filesystem forensics confirmed it was genuinely progressing, not deadlocked: the target tree still held 134,186 files (130,633 of them under node_modules), with node_modules mtime 02:13 and the package store mtime 02:14 — i.e. actively unlinking mid-poll. Statements 3–7 of the compound never ran.

Per-turn bills during the loop (token_count events):

rec  trigger  input     cached   output  total
662  wait     237,280   236,672      31  237,311
665  wait     237,339   236,672      31  237,370
668  exec     237,400   236,672      45  237,445
...
931  wait     242,794   241,792      31  242,825
937  exec     242,920   241,792     396  243,316

90 turns: input 21,610,789 / output 3,837
mean:     240,119 in / 42 out     ratio 5632:1

Input ratchets upward monotonically (237,280 → 242,920) because each poll writes itself into the history that the next poll must re-send. Every turn's entire product is 31 tokens: {"cell_id":"89","yield_time_ms":1000,"max_tokens":5000}.

Inter-call gaps are irregular inference latency, not a schedule — median 3.05 s, range 2.23–15.27 s (wait-preceding, n=72). There are zero schedule_wakeup, user_message, task_started or interrupt records after 648. And with yield_time_ms: 1000, the 67 waits spent a combined 67 seconds actually waiting, across 9.5 minutes of wall clock.

The escape hatch exists, and nobody can use it

I want to be precise here, because the fix is not "add a primitive". The primitive is already shipped:

// @exec: {"yield_time_ms": 120000, "max_output_tokens": 1000}

The CLI's own documentation even demonstrates it: "imagegen needs a few minutes to finish. In code-mode, use the first-line @exec directive to give the initial call 120 seconds and the same yield for any waits that follow." There is no documented upper bound — the only stated constraint is "must be non-negative safe integers".

Across 19,392 exec calls in my fleet's rollouts, the pragma was used 0 times.

That is the actual defect surface:

  • The default (10000 ms) is far below the duration of ordinary filesystem and build operations.
  • Exceeding it degrades silently into a paid polling loop instead of surfacing a handleable condition.
  • Using the escape hatch requires knowing the command's duration before running it — precisely the thing you cannot know for rm -rf, bun install, git worktree remove, or any user-supplied tool.
  • Resumption is pull-only, so the cost of not knowing is billed per turn at full context.

This is not a one-off

Same fleet, same rollout corpus:

  • 8,752 MCP tool calls; 86 took ≥10 s; 70 of those 86 (81%) were parked into a cell. Each parked call then needed median 4 / mean 4.7 / max 13 wait turns to collect. 329 waits + 70 parked execs ≈ 399 turns of pure polling — on the order of 10^8 tokens at these context sizes.
  • All 12 slowest calls were a single long-running tool endpoint. Any user who registers a slow MCP tool or shells out to an external CLI reproduces this by default.
  • Global wait yield_time_ms distribution: 30000 → 2,516 calls, 1000 → 979 calls. The model frequently picks a 1-second poll interval against multi-minute work, which is the worst possible ratio. (I'll note honestly that this specific choice is model behavior — but the CLI is what makes a 1 s interval cost 240k tokens instead of being harmlessly cheap.)

Question about quota enforcement

In the same session, rate_limits.primary.used_percent progressed:

17:40:07   21,664 cumulative tokens    92.0%
17:46:01    4,176,704                  95.0%
17:53:23   14,271,350                  98.0%
17:57:08   20,084,287                 100.0%   <-- limit reached

The busy-wait loop did not begin until ~18:05 — 8 minutes after used_percent hit 100.0 — and the CLI accepted a further 18 minutes and ~34.6M tokens of requests, with rate_limit_reached_type reported as null on every event.

Is post-100% execution expected behavior? If the limit is a soft accounting boundary rather than an admission gate, that is a reasonable design — but combined with the polling loop above it means a single unattended slow command can spend multiples of a weekly allowance with no backpressure anywhere in the stack.

Expected behavior / asks

  1. Push cell completion, or at minimum make the default yield adaptive (e.g. exponential backoff up to a bounded ceiling) so that N-minute commands do not cost N/interval full-context turns.
  2. Raise the 10 s default, or apply the documented pragma value to follow-up waits automatically.
  3. Include elapsed time and a liveness signal in the wait/park output so the model can distinguish "progressing" from "hung" and choose a sane interval. Right now both look like empty output.
  4. Cap or warn on repeated identical polls — 16 byte-identical write_stdin calls in a row is trivially detectable.
  5. Per-session usage telemetry surfaced to the user, so a runaway loop is visible before it consumes a plan rather than after.
  6. Quota reset for the affected account. The 34.6M tokens were spent entirely after the task's result had been delivered and its artifact written, in a loop the model had no primitive to avoid and no signal to exit.

Environment

  • macOS (Darwin 25.3.0), Apple Silicon
  • Codex CLI in code mode, ChatGPT-plan auth, plan prolite
  • Non-interactive codex exec, multiple concurrent sessions
  • Evidence: rollout JSONL 019ffc35-8a14-76b3-bd0b-7e6bfe18b62c (identifiers redacted; happy to share the relevant token_count / custom_tool_call / function_call records on request)

Related

Activity

  1. added
    bugSomething isn't working
    CLIIssues related to the Codex CLI
    execIssues related to the `codex exec` subcommand
    tool-callsIssues related to tool calling
    rate-limitsIssues related to rate limits, quotas, and token usage reporting
    on Aug 14, 2026
  2. jdcodes1 commented on Aug 18, 2026

    @jdcodes1

    Verified the shape on main @ 1f41cc5: wait (core/src/tools/code_mode/wait_spec.rs) takes a model-chosen yield_time_ms, there's no completion push, and nothing in the tool result nudges the model toward longer yields — in your incident it chose 1,000 ms polls, and the harness faithfully executed 90 of them.

    Three fix layers, smallest first:

    1. Server-side minimum/escalating yield: when wait is called on a cell that's still running and produced no new output, the harness can simply block longer than requested (escalating 30s → 2m → 5m) before returning — turning 90 poll turns into ~3 without any protocol change. The model's request is a hint, not a contract worth 240k tokens per second of impatience.
    2. Completion push: deliver "cell N finished (exit 0)" as an injected item at the start of the next turn, so a finished command never needs a poll turn at all.
    3. Worth checking in your rollout: whether those 90 near-identical poll turns hit prompt caching. 21.6M billed input for resubmitting the same context suggests they didn't — if so, that's a separate, equally large bug.
  3. jmtt89 commented on Aug 27, 2026

    @jmtt89

    Second independent incident with the same mechanism. Two things to add: a data point on your caching question, and a second cost profile for the same bug.

    Codex CLI 0.149.0, codex exec driven by an orchestrator (non-interactive). Our variant is write_stdin polling on a unified_exec session rather than wait(cell_id) in code mode — same pull-not-push shape, different tool.

    Session totals (175 turns):

    input tokens 23,957,294
    cached input 23,542,272 (98.3%)
    new input 415,022
    output tokens 52,694
    input:output 454:1

    174 tool calls: 86 write_stdin, of which 84 are pure polls (chars: ""), against 80 exec_command. Requested yield_time_ms across all 166 occurrences: {1000: 68, 10000: 40, 25000: 1, 30000: 56, 35000: 1} — note the 35000, where the model asked for 35s and clamp_yield_time silently capped it to 30s (#22541).

    Splitting turns by kind is where it gets clear:

    poll turns work turns
    turns 84 90
    context re-sent 13,507,179 10,450,115
    new input 179,819 235,203
    output 6,151 46,543
    avg context/turn 160,799 116,112
    avg output/turn 73 517

    The polls re-sent more context than all the real work combined, for 12% of the output.

    On your point 3 (caching): ours hit cache, and it did not save us. 98.7% on the poll turns alone — 13,327,360 of 13,507,179. Your 21.6M billed input for 90 near-identical polls suggests you did not get those hits.

    But per #37299, cached context is still metered against the subscription: that report shows the account usage API returning ~290M tokens for a single day at 97–98% cache, taking a Pro account from 0% to 90% of its weekly limit in ~15.5 hours. Their per-turn profile (~137–141k input, 97.4–98.1% cached) is almost identical to ours.

    So a cache hit makes this loop cheaper in dollars but not in quota. Our 13.5M of re-sent poll context counted against the window even though only 179,819 tokens of it were new. Worth stating explicitly in the fix discussion: layers 1 and 2 pay for themselves in quota terms even when prompt caching works perfectly. A cache-only fix would not.

    A second cost profile, which I think broadens the report. You noted that git worktree remove prints nothing while unlinking. We were waiting on CI with gh run watch, which without a TTY cannot overwrite with ANSI, so it reprints the entire job tree on every 10s refresh. That put 179,819 tokens of genuinely new input through the loop on top of the context re-send — 2,140 per poll, actually more per turn than our work turns' 2,613.

    So the same bug has two shapes: a silent command costs pure context re-send, a chatty one costs that plus real input. Both reduce to "asking whether the process is done costs the same as doing the work".

    And one nuance: our wait was entirely legitimate. The agent had finished, committed, pushed, opened a PR, and was waiting on a pipeline that genuinely took 21.6 minutes — jobs running sequentially with multi-minute gaps. It wasn't spinning on a hung process and had no way to know better. So this isn't only about commands that produce no output; it also hits the ordinary "wait for my own CI" pattern, where the correct duration is tens of minutes and the harness caps observation at 30s.

    Your fix layer 1 (escalating server-side yield) would have collapsed our 84 polls to roughly 5. Layer 2 (completion push) would have made them zero.

    Workaround for anyone hitting this from an orchestrator: instruct the agent not to wait for CI at all — finish the turn, and let the caller poll outside the model loop, where a gh run watch subprocess costs nothing. And if an agent must wait on something, prefer a command that stays silent until it's done: the redraw is what turns a cheap loop into an expensive one.


    Cross-linking, since these two threads don't reference each other: #37299 has the billing-side evidence (account usage API, ~290M/day at 97–98% cache) that answers the caching question raised here, and this issue has the mechanism-level explanation that #37299 lacks. The automated duplicate detector flagged them as possible duplicates — they are not; they are two halves of the same problem, and a fix for one does not fix the other.

  4. Razorex001 commented on Sep 14, 2026

    @Razorex001

    This happens also in the ChatGPT app, it's the most annyoing thing. Sooo much usage credits burned on sol/astra that continuously check the build progress incredibly often, for no reason other than telling me "how it's going", like "okay the build has entered xyz phase, this this and that happened so the next part is the abc part". Why would I need / want to use my usage credits on narration of the build progress? Just check when it finishes or encounters a critical error that doesn't let it proceed.
    Pretty sure OpenAI could save nearly 30% of their inference compute capacity, just by fixing this.

  5. turtlenosleep commented on Sep 15, 2026

    @turtlenosleep

    Independent confirmation from a newer Codex rollout.

    Environment:

    • macOS, Apple Silicon
    • affected rollout: codex-cli 0.153.0, code mode

    In one model turn lasting about 24 minutes, the rollout recorded 42 functions.wait calls:

    • 22 × yield_time_ms: 1000
    • 20 × yield-time_ms: 10000
    • 8,325,652 input tokens (8,152,320 cached; 173,332 uncached)
    • 1,917 output tokens

    Each wait was followed by a distinct token_usage_record / response_id, confirming that an unchanged pending state re-enters the model rather than being handled as a client-side or event-driven wait.

    Across the same rollout, 147 wait calls could be paired with usage records:

    • 24,800,609 input tokens
    • 24,423,680 cached input
    • 376,929 uncached input
    • 6,538 output tokens

    I also checked current openai/codex main. DEFAULT_WAIT_YIELD_TIME_MS remains 10,000 ms, and the model-facing wait handler passes the requested interval into YieldAfter; the runtime has an internal wait_to_pending path, but it is not exposed as the model-facing wait tool.

    This supports the proposed fix: event-driven/push completion, or at least harness-side blocking/backoff that does not create a new model response while state is unchanged.

    No prompts, session identifiers, repository names, local paths, or account details are included above.

  6. mkXultra commented on Sep 16, 2026

    @mkXultra

    I implemented Cell-completion wake in a personal fork, addressing the completion-push part of this report for the local interactive CLI. This is not a fix landed upstream, and it does not resolve the report's non-interactive codex exec case or its other quota/telemetry requests.

    A yielded Cell can finish while the model is idle. Its completion queues a bounded Cell-ID notification, then starts a continuation only when the thread is idle. The continuation uses the existing wait(cell_id) to retrieve retained Cell output; wake does not inject unlimited nested-tool output. Busy turns are not interrupted or steered. Waiting for the completion event itself requires no model polling.

    There was also an independent blocker: a live yielded Cell's second nested tool call could stop progressing at idle because the dispatch worker was scoped to a sampling request. The fork now retains dispatch while unfinished Cells exist, with teardown and approval-admission handling. I reported that separately in #45945.

    Local Linux smoke test: one Cell awaited an 8-second command and then another 8-second command. The assistant ended its turn after the outer Cell yielded at one second. Both tools completed, the Cell triggered a continuation, and one post-notification wait returned both explicitly printed results. Deterministic sequential HTTP MCP regression coverage also exercises dispatch at idle before any new model request. Tools requiring new approvals/user input, and notify, can still need a subsequent active turn; ordinary tool timeouts still apply.

    Reference implementation: Cell completion wake, idle Cell dispatch fix.

    Trying the fork

    Successful GitHub Actions build and downloads, built from commit af1f26f725:

    These are unofficial fork artifacts, not an upstream release. GitHub sign-in may be required to download them; the current artifacts expire on October 15, 2026. Extract the package inside the downloaded artifact and keep its bin directory together, including codex-code-mode-host and the other bundled helpers. Run bin/codexg (bin/codexg.exe on Windows). The fork uses the normal Codex configuration/authentication/history by default. Keep the bundle in its own directory to avoid replacing an installed upstream CLI or its helpers.

    All three builds/package checks passed. End-to-end wake testing was performed locally on Linux; CI success is not a claim of full wake-behavior testing on macOS or Windows.

  7. rezzminator commented on Sep 21, 2026

    @rezzminator

    Fleet-scale confirmation, plus the exact clamp and a config workaround we verified today.

    Environment: codex-cli 0.154.0 (macOS, Apple Silicon) and 0.153.4 (Linux), code mode, gpt-5.6-sol and gpt-6-astra, multi-agent flights (an orchestrator thread spawning sub-agents). All numbers below are measured from the rollout files (token_count events deduplicated by cumulative total), not estimated.

    Cost, one night

    • 201 sub-agent threads, 6,945 model calls, 755M input tokens, 3.0M output tokens (250:1); 96% cached. This consumed roughly 40% of a weekly limit.
    • 1,387 model calls were an empty write_stdin({chars:""}) issued for no reason other than "is it done yet": 185.5M input tokens, 25% of all sub-agent input, at an average context of 133–138K per poll. Another 229 wait/sleep calls: 28.9M.
    • Requested yield_time_ms on those polls: 30 s × 883, 60 s × 71, 300 s × 96, 600 s × 15. Poll chains up to 30 calls long for one test run.
    • The orchestrator threads added ~70 wait_agent calls each at 100–143K context.

    The clamp, measured

    Grouping every tool call by requested yield_time_ms and reading Wall time from its output (n ≈ 2,700 calls):

    requested yield cell has // @exec pragma longest wall time actually held
    30 s no 31.0 s
    60 s no 31.0 s
    120 s no 31.0 s
    300 s no 31.0 s
    600 s no 31.0 s
    120 s yes (yield_time_ms ≥ inner) 120.0 s
    300 s yes 300.0 s

    So the inner tool's yield_time_ms is silently capped at ~30 s by the outer exec cell unless the script opens with a // @exec: {"yield_time_ms": N} pragma. The model asks for 600 s, gets 31 s, is never told its request was reduced, and concludes it must poll. The tool description says "Defaults to 10000 ms" and documents no ceiling. This is the same cap reported in #22541, and it is the reason the loop in this issue is so dense: even a model that tries to wait properly cannot.

    Workaround we verified (config only, no fork)

    background_terminal_max_timeout = 3600000
    
    [features.code_mode]
    default_exec_yield_time_ms = 3600000

    A/B with codex exec, the same natural-language request ("run this ~100 s command and tell me the result"), nothing else changed:

    • without: write_stdin asked 60 s → held 31.0 s, asked 50 s → held 31.0 s; 6 model calls, 150K tokens.
    • with: write_stdin asked 60 s → held 60.0 s.
    • with the config plus one line of developer instructions ("wait with ONE write_stdin sized to the expected duration; long yields are honoured"): a 200 s command = exec_command (1.2 s) + one write_stdin held 196.5 s, zero polls; a following quick command was unaffected.

    Also verified: a single cell can loop tools.write_stdin in JavaScript until exit_code appears, under a pragma — 75 s command, one model call, both polls inside the script at zero token cost.

    Hooks cannot mitigate this today

    We tried to guard it with hooks. PreToolUse/PostToolUse fire for the nested exec_command (surfaced as tool_name: "Bash", tool_input.command) and only once, when the command completes. They never fire for write_stdin, wait or wait_agent, and the payload carries neither the cell source nor its yield, so a hook can neither see a poll nor deny it. Relevant to #39596.

    What would fix it

    1. Honour the requested yield_time_ms up to background_terminal_max_timeout, or return the effective yield in the tool result so the model learns its wait was shortened. A silent cap is the defect.
    2. Completion push: a yielded cell that finishes wakes the thread (the fork described above), including in codex exec and in sub-agent threads.
    3. State the ceiling in the exec / write_stdin / wait tool descriptions, and ship a sane default above 30 s for cells that are only awaiting.
    4. Fire hooks on write_stdin / wait / wait_agent, with the cell source in the payload.
    5. Surface poll turns in usage telemetry: a turn whose only action is an empty write_stdin or wait is distinguishable and should be visible as such.

    Happy to share the (anonymised) measurement scripts; they read only token_count, custom_tool_call and custom_tool_call_output records.

  8. titanslaw commented on Oct 15, 2026

    @titanslaw

    A current source and rollout audit confirms this code-mode boundary remains relevant. codex-rs/core/src/tools/code_mode/wait_spec.rs defines model-facing functions.wait; wait_handler.rs passes each wait to code_mode_service.wait(...) and returns the result to the model. Current CODEOWNERS assigns /codex-rs/core/ to @openai/codex-core-agent-team.

    Focused reproducer: hold one yielded cell through three intervals with no output or material event, and count response IDs that only request another wait on the same cell. Replay matched cases for output, completion, error, user interruption/cancellation, concurrent worker feedback, and a due report/heartbeat. The target is zero model responses solely for repeat-empty intervals, while every substantive event and due action remains timely. Please compare before/after using the same scenario and response-ID method and report actual counts.

    Please route to the owner of native pending-cell continuation and model-turn wake, including non-interactive codex exec and sub-agent execution. Existing cancellation, concurrent feedback, and due updates must remain interruptible.

  9. titanslaw commented on Oct 15, 2026

    @titanslaw

    Additional source context for the native owner: merged #48123 added a preempt signal to code-mode execute/wait so an observation can yield early while its cell stays live, with in-process and remote-transport coverage. Merged #48135 uses that path for queued user input behind the opt-in instant_interrupt flag and tests repeated steering during exec/wait, retained tool results, and later collection. A pending-result wake change should reuse/preserve these interruption semantics and should not turn an empty interval into a new model response. Neither merged PR establishes completion-driven model wake or the matched before/after response count requested above.

  10. titanslaw commented on Oct 15, 2026

    @titanslaw

    Further source narrowing for the platform owner: in current codex-rs/core/src/tools/code_mode/delegate.rs, CodeModeCellDelegate::cell_closed only calls broker.close_cell(cell_id). That broker method removes the dispatch gate and finishes cell recording; it does not schedule an idle model continuation. The runtime calls the delegate on cell closure (code-mode-runtime/src/session_runtime/mod.rs), and the shared CodeModeSession trait (code-mode-protocol/src/session.rs) exposes interruptible execute/wait but no pending-to-completion subscription to core. The in-process service has wait_to_pending, while model-facing wait_handler.rs uses the timed wait path. This places the missing completion-to-idle-turn bridge at the core delegate/session turn boundary plus the cross-transport session contract. Proposed owner behavior: retain the cell result, queue at most a bounded cell-ID wake on closure, resume only when the turn is idle, and preserve queued user input, cancellation, concurrent feedback, and due reports. Please validate this owner mapping; it is a source inference, not a claim that one callback edit alone is sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIbugSomething isn't workingexecIssues related to the `codex exec` subcommandrate-limitsIssues related to rate limits, quotas, and token usage reportingtool-callsIssues related to tool calling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions