Skip to content

high-frequency code-mode polling, amplified by a very large resumed context #32309

Open

Description

@ssh4net

After consiming unusual huge tokens usage with a Codex Sol 5.6 ~600mln tokens for a day vs usual 150-200mln on a same code base and same sessions from Codex 5.5 i passed a logs to a Codes to understand if this is not my own WSL bug fix (#16690) that force me to build a codex-cli every wheek with latest upstream fixes.

And Codex 5.6 Sol xHigh found a possible culprit (maybe this is a known issue?):

During the final Medium interval:

  • 43 minutes, 2 user messages
  • 292 model responses
  • 96 exec calls and 192 wait calls
  • All 192 waits explicitly used yield_time_ms: 1000
  • 49.39M raw tokens
  • Only 10,828 reasoning tokens
  • Five-hour usage increased from 58% to 100%
  • Cache ratio was healthy at 98.3%

The CLI documents a 10-second default (codex-rs/core/src/tools/code_mode/wait_spec.rs:13), but accepts the model’s one-second value unchanged (codex-rs/core/src/tools/code_mode/wait_handler.rs:92). There is no minimum clamp. Every poll causes another model response with approximately 170K–240K input tokens.

Other Findings
The parent rollout now totals 1.153B raw tokens, with 95.4% cached. Most historical usage was GPT-5.5 xhigh (967M), but Sol added 268M on July 10 alone.

Secondary contributors:

  • One guardian auto-review used 60,882 additional raw tokens. approvals_reviewer = "auto_review" is enabled.
  • Historical view_image outputs occupy 132.6 MB, including 48 outputs over 1 MB.
  • Multi-agent is enabled but no actual spawn_agent activity appears.
  • There were no code-mode-host spawn failures in the parent thread.
  • No goals or automatic goal continuations were detected.

On time of investigation upstream still don't have a fixes.

Activity

  1. added
    bugSomething isn't working
    CLIIssues related to the Codex CLI
    tool-callsIssues related to tool calling
    contextIssues related to context management (including compaction)
    rate-limitsIssues related to rate limits, quotas, and token usage reporting
    on Jul 11, 2026
  2. github-actions commented on Jul 11, 2026

    @github-actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  3. MickaelOth commented on Jul 12, 2026

    @MickaelOth

    I can confirm a very similar pattern on Codex app on Windows, mainly using GPT-5.6 Sol with medium reasoning.

    An otherwise ordinary component task reached 117,961,931 raw tokens across 971 model responses while running tests. It made 683 wait calls, including 305 with yield_time_ms: 1000. The input context had a median size of ~129K tokens and reached 217K, with ~99% cache utilization, while producing only ~96K output tokens and ~22K reasoning tokens.

    This does not appear to be isolated. In my July 9–11 local rollouts:

    • 10 threads with at least 10 wait calls accounted for 458.7M raw tokens and 3,901 model responses.
    • 6 threads without significant polling accounted for only 3.9M raw tokens and 77 responses.
    • All 8 threads exceeding 10M tokens showed frequent polling around potentially long-running commands, particularly Playwright tests.

    During full e2e suites lasting 7–9 minutes, I observed 35–42 polling rounds per run, generally using 10-second waits. Later targeted runs frequently used one-second waits. Each completed wait appears to cause another model request with a large, mostly unchanged context.

    For additional context, from July 1–8 only 3 of 22 active threads exceeded 10M raw tokens, and those were genuinely large, highly active tasks. From July 9–11, 8 of 16 exceeded 10M, including otherwise ordinary tasks spending time waiting for long-running commands.

  4. khrm355 commented on Jul 12, 2026

    @khrm355

    Sampled some usage i have done with 5.5 and after 5.6sol, seems like this existing problem amplified with 1M Context size increase.

    Image
  5. khrm355 commented on Jul 13, 2026

    @khrm355

    It seems like openai team also on track, after these changes they have done i will do the same analyses again. Context size change probably lowered the impact but not an solution.

    Updates for Codex and ChatGPT Work users. No nerfing, only good stuff!

    - We have landed inference optimizations and are passing down savings to all the subscriptions for GPT-5.6 Sol. That should result in around 10% more usage on its own.
    - We noticed that by changing the… https://t.co/ej7DdJeyBL

    — Tibo (@thsottiaux) July 13, 2026
  6. bbingz commented on Jul 25, 2026

    @bbingz

    Analysis (community)

    Residual analysis for #32309 (code_mode wait polling re-bills the full transcript every yield) under docs/contributing.md — invitation-only. No unsolicited PR.

    Root-cause hypothesis

    The code_mode wait tool has no minimum clamp at any layer on yield_time_ms, and every yield re-enters the model with the full growing prefix, so a long-running exec cell becomes N ≈ ceil(runtime / yield_time_ms) billable sampling requests — the 1s-poll variant of the re-entry mechanism.

    @ssh4net — your forensic work here is the sharpest trace of this burn I've seen: 192 waits at yield_time_ms: 1000, 170K–240K input tokens per re-sample, and the 1.153B raw-token parent rollup isolate the mechanism precisely. I want to add the missing piece: the full re-entry chain from wait_spec.rs through the turn layer, plus a code_mode-specific fix outline that the wait_agent analysis on #35259 explicitly did not cover.

    What #35259 covered vs what this is. The analysis posted on #35259 traced multi_agents_v2/wait_agent (codex-rs/core/src/tools/handlers/multi_agents_v2/wait.rs:178-196) — a different handler that polls a subagent mailbox via InputQueue::subscribe_activity, with a 30s default / 10s floor / 1h max, and out-of-range timeout_ms rejected at wait.rs:54-63. That post explicitly noted its fix "doesn't cover code_mode wait or explicit short choices." This is that uncovered path. The code_mode wait tool (codex-rs/core/src/tools/code_mode/wait_spec.rs + wait_handler.rs, tool name "wait") polls an exec cell (process stdout), uses arg yield_time_ms (not timeout_ms), defaults to 10s (not 30s — code-mode-protocol/src/runtime.rs:12 DEFAULT_WAIT_YIELD_TIME_MS), and has no config knob (the default is a flat const; compare exec's BUFFERED_EXEC_YIELD_TIME_MS override at core/src/tools/code_mode/mod.rs:57).

    The no-min-clamp finding, confirmed and extended. You already cited wait_spec.rs and stated "there is no minimum clamp." Confirmed at HEAD 4c43465133, with the structural reason and the full chain:

    1. Schema layer — structurally impossible. The JsonSchema struct (codex-rs/tools/src/json_schema.rs:41-74) has no minimum/maximum fields; JsonSchema::number() emits only {"type":"number","description":...}. A clamp cannot be expressed in the tool schema the model sees.
    2. Handler layer — pass-through. wait_handler.rs:94 copies yield_time_ms: args.yield_time_ms straight into WaitRequest with zero comparison.
    3. Runtime layer — no floor. code-mode/src/service.rs:35-42 yield_timeout(): the +1s grace (from Add grace period to code-mode yield timeouts #33867) applies only for yield_time_ms >= 10000 (MIN_YIELD_TIME_FOR_GRACE); any sub-10s value returns the raw Duration::from_millis(value), which reaches cell_actor/mod.rs:569 tokio::time::sleep(duration) — the cell sleeps exactly 1s then yields.
    4. Re-entry — full prefix re-ingestion. wait returns an ordinary FunctionToolOutput (code_mode/mod.rs:213, success flag Some(true)). stream_events_utils.rs:324 sets needs_follow_up = true; turn.rs:2199 aggregates it and turn.rs:2369-2371 fires on end_turn=false; the loop re-invokes run_sampling_request with sess.clone_history().for_prompt(...) (turn.rs:295-301) — the model re-reads the entire growing transcript of prior wait outputs just to emit the next 1s wait call. That is the 170K–240K per poll.

    So: N = ceil(runtime / yield_time_ms) billable re-samples, each billing the full prefix. With yield_time_ms=1000, roughly one full-prefix inference per second for the life of the cell.

    Ranked fix outline (all OSS):

    Related cluster: #32640 observed this same burn empirically but attributed it to multi_agent_v2 — the params {cell_id, yield_time_ms, max_tokens} in its logs are the code_mode wait signature, not wait_agent; #29122 (code_mode silent activation); #33669 (tool_mode=direct override).

    Fail-first tests I'd want before any merge: (1) yield_time_ms=1000 is clamped to the floor in the WaitRequest the service receives (today: passes 1000 through unchanged); (2) a 10-min cell yielding every 1s produces ≤1 run_sampling_request for the whole wait (today: ~600); (3) yield_timeout(1000) returns ≥ the floor (today: returns raw 1000ms). Happy to draft the test patch if a maintainer wants it.

    Happy to refine measurements or the fail-first tests if maintainers want them. No unsolicited PR from me.

  7. jkammerland commented on Aug 2, 2026

    @jkammerland

    Trying to find relevant issue here. But I still experience codex has a strong preference for ending up in polling loops, sometimes for no reason. @1.46.0

    Image

    This is one of the more obvious one, but codex will often end up in polling loops for longer builds, checking test status in other machines etc.

    This eat tokens like crazy, I assume. I do feel like I run out very quickly when this problems happens.

  8. bbingz commented on Aug 6, 2026

    @bbingz

    @jkammerland — to place your 1.46.0 case correctly, the params matter, because two different handlers produce near-identical symptoms. If the tool call is wait with {cell_id, yield_time_ms, ...} it is the code_mode path traced above. If it is wait_agent with timeout_ms it is the multi-agent path, which already has a 10s floor and rejects out-of-range values (codex-rs/core/src/tools/handlers/multi_agents_v2/wait.rs:54-63). #32640 hit this same burn and attributed the code_mode signature to multi_agent_v2, so the distinction is worth nailing down before more measurements pile up.

    For the code_mode path nothing has changed on current main: codex-rs/core/src/tools/code_mode/wait_handler.rs:106 still copies yield_time_ms: args.yield_time_ms into WaitRequest with no comparison, so a model-chosen 1s yield still reaches tokio::time::sleep unclamped and re-bills the entire growing prefix roughly once per second. Fix (a) above — a floor at the handler layer, which is where it has to live since JsonSchema cannot express minimum (codex-rs/tools/src/json_schema.rs:41-74) — remains the smallest diff that closes the 1s vector. Happy to draft the fail-first tests if a maintainer wants them.

  9. kirillsaven commented on Aug 12, 2026

    @kirillsaven

    I replayed four long-lived Windows Desktop parent rollouts (Aug 7-12, Asia/Bangkok) against several circuit-breaker policies.

    The sample contains about 27.6k outer exec calls and 7.4k outer wait calls; 26.7k exec cells contained exactly one nested tool, while only about 700 contained multiple tools. There were 349 wait bursts, 113 longer than 10 calls, with a maximum of 500.

    A third-wait boundary is the better default: blocking/coalescing the second wait catches only about 941 additional historical calls but also affects all 44 ordinary two-wait bursts. An adaptive "second only for <=5s waits" policy adds just 22 catches over the third-wait policy. A separate cap of 20 same nested-tool roundtrips per 10 minutes would have intercepted about 7.7k calls; cap 40 only about 1.2k.

    There is an important mitigation gap on current main (0e82c62a449c484edc053fa3ff82410870df0145): CodeModeWaitHandler::pre_tool_use_payload() explicitly returns None, and CodeModeExecuteHandler receives a custom payload, so the default function-tool PreToolUse contract does not apply. Therefore user hooks cannot guard the outer exec/wait model-roundtrip boundary; they can only see many nested tools. A local nested-tool cap helps because more than 96% of sampled exec cells were single-tool cells, but it cannot stop a pure outer wait loop.

    Suggested runtime fix:

    1. Coalesce repeated yielded waits locally and re-enter the model only on output, exit, user input, or a configurable progress interval.
    2. After two no-progress wait yields, keep the third and later waits local rather than sampling again.
    3. Add a progress-aware per-turn model-follow-up budget, defaulting to a warning/aggregation request after 20 single-tool follow-ups in 10 minutes.
    4. Expose this follow-up boundary to hooks/config, rather than relying on nested PreToolUse.

    This complements the existing minimum-clamp proposal and avoids penalizing legitimate two-stage waits.

  10. bbingz commented on Aug 20, 2026

    @bbingz

    Analysis (community)

    @kirillsaven — the Windows Desktop parent-rollout sample (27.6k outer exec, 7.4k outer wait, many 1-nested exec cells) is useful because it separates code_mode wait from multi-agent wait_agent.

    Root-cause hypothesis (re-checked 2026-08-20 at main 3b45c2906): the two handlers are not equivalent on timeout policy.

    • code_mode wait still has no minimum clamp. yield_time_ms is deserialized and passed straight into WaitRequest (codex-rs/core/src/tools/code_mode/wait_handler.rs:27-28,104-106). The tool spec documents a default of 10000 ms (wait_spec.rs:13-16) but does not enforce a floor. Short yield_time_ms (e.g. 1000, as in earlier reports on this issue) still re-enters the model with the growing prefix.
    • multi-agent wait / wait_agent does clamp (codex-rs/core/src/tools/handlers/multi_agents/wait.rs ms.clamp(MIN_WAIT_TIMEOUT_MS, MAX_WAIT_TIMEOUT_MS); v2 path also emits a clamped-timeout message).

    If the 7.4k outer waits in your sample are code_mode wait (cell_id + yield_time_ms), they sit on the unclamped path this issue tracks. If they are wait_agent (timeout_ms on a child agent), they are the D/#35259 family (local timeout → follow-up turn), not a missing yield clamp.

    I am not treating the 27.6k exec volume as a defect by itself — only the wait/re-entry policy. No unsolicited PR. A fail-first test for this issue is: code_mode wait with yield_time_ms: 1 is raised to a documented minimum before code_mode_service.wait.

    Issue anchor remains #32309. Happy to split a table of wait-tool names vs clamp behavior if that helps triage incoming Desktop samples.

  11. kirillsaven commented on Aug 28, 2026

    @kirillsaven

    @bbingz Thanks for calling out the handler distinction. I rechecked the retained replay script and need to correct the interpretation of my earlier aggregate.

    Its WAIT_TOOLS set combines wait, wait_agent, wait_threads, and codex_app__wait_threads. Both the aggregate labelled outer_wait_calls_bypass_pretooluse and the wait-burst calculation use that combined set. That label was too broad: the aggregate is not valid evidence that every counted wait uses the unclamped code_mode path.

    A fresh run over the four currently retained parent journals, with the script's explicit window [2026-08-07, 2026-08-13) at UTC+7, gives this tool-name split:

    Tool name Calls
    wait 3,418
    wait_agent 384
    wait_threads / codex_app__wait_threads 0
    exec 14,591

    This does not reproduce the approximately 7.4k waits / 27.6k execs in my previous comment. I cannot reconcile that earlier total from the currently retained inputs, so please do not use it as a code_mode-only denominator or as a verified estimate of billing savings. The earlier second/third-wait comparisons were also based on the combined wait set and should not be treated as handler-specific measurements.

    The narrower runtime point is separate: the code_mode wait handler's absent PreToolUse payload in the pinned source I linked means a nested-tool hook cannot guard that outer model-follow-up boundary. I agree that code_mode yield clamping and multi-agent timeout-to-follow-up behavior need separate tests/issues. A minimum clamp plus output/progress-aware wait coalescing is the relevant code_mode proposal; raw exec volume alone is not proof of a defect.

    No new claim about current-main clamp behavior is being made here, and no unsolicited PR is being opened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIbugSomething isn't workingcontextIssues related to context management (including compaction)performancerate-limitsIssues related to rate limits, quotas, and token usage reportingtool-callsIssues related to tool calling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions