Desktop wait/status orchestration re-meters full cached context every 10-30s; stale 'running' subagents keep it looping (90% of weekly Pro in 15.5h) #37299

Activity

1. John-Lin98 commented on Aug 13, 2026

   

   John-Lin98

   on Aug 13, 2026 – with ChatGPT Codex Connector

   More actions

   Follow-up measurement from one Windows Codex Desktop installation (anonymized; no session text, IDs, paths, account data, or rollout excerpts).

   Method:

   • Aggregated only local JSONL tool-event metadata, wait arguments, and the local token_count counters.
   • Excluded the active session.
   • Split sessions at the time a global long-wait instruction was added: functions.wait >= 180000 ms, with 300000 ms preferred when no intermediate output is needed.

   Observed local counts:

   • Before the policy: 2 sessions, 161 functions.wait calls; 158 were <180 s and 82 were <30 s. Local counters totaled 78.2M tokens (74.8M cached input).
   • After the policy: 10 sessions, 262 functions.wait calls; 80 were <180 s and 40 were <30 s. All 80 short waits came from one session; the other 9/10 sessions had zero short functions.wait calls. There were also 12 wait_agent and 6 list_agents calls. Local counters totaled 863.5M tokens (834.7M cached input).

   Interpretation limits:

   • The before/after workloads, session lengths, and contexts are not comparable, so this does not estimate token savings or prove billing attribution.
   • These are client-side local counters, not server-side telemetry.
   • The instruction-level workaround substantially changed the observed functions.wait pattern in most new sessions, but one post-policy session still looped at 1–30 s. That makes an enforced client/server no-change long-wait primitive important: guidance alone is not a reliable guardrail.

   This independently supports the request that no-change timeout/status results should avoid waking a full model turn, and that wait_agent / list_agents need the same treatment as code-cell waits.
2. GorangN commented on Aug 21, 2026

   

   GorangN

   on Aug 21, 2026

   More actions

   I can reproduce a very similar pattern on Codex app-server 0.148.0-alpha.9 with GPT-5.6 Sol xhigh and multi-agent.

   In my case, a single multi-agent session exhausted essentially 100% of my weekly Codex allowance in about two hours.

   Observed timeline:
   • ~0% weekly usage at 19:37 CEST
   • ~50% at 20:56
   • ~80% at 21:19
   • ~95% at 21:34
   • 100% at approximately 21:40

   Parent thread:
   01a0203f-10c5-7431-998e-c2643c4220ab

   The parent spawned six subagents across two waves. All inherited GPT-5.6 Sol + xhigh.

   After deduplicating forked/replayed token_count history, my local rollout analysis indicates approximately:
   • 174M total processed tokens
   • 169M cached input tokens
   • ~97–98% cached input on many turns
   • ~135k average input tokens per parent model turn
   • 8 of 9 observed wait_agent calls timed out

   I can also see cases where a wait_agent timeout is followed by another model turn processing 100k–190k+ input tokens, almost entirely cached.

   This looks extremely similar to the re-metering behavior described in this issue.

   I have retained the parent and all six subagent rollout JSONLs and can provide sanitized excerpts if useful.
3. lunara69-ctrl commented on Aug 24, 2026

   

   lunara69-ctrl

   on Aug 24, 2026

   More actions

   Independent data point from a Linux Codex CLI audit:

   I observed the same qualitative pattern, although at a much smaller scale than reported here.

   Across one audited workflow:

   • 58 no-new-information WAIT/POLL model continuations
   • 6,787,563 total input tokens attributable to those continuations
   • ~117,027 input tokens per WAIT/POLL continuation on average

   The larger session accumulated 55.9M total MAIN input tokens, with repeated context processing becoming a major component of usage.

   One additional observation: cached input changes the economics, but does not remove the architectural amplification. A semantically empty status check can still cause a very large existing context to be processed again.

   So I would strongly support exposing telemetry for:

   1. logical wait operations,
   2. parent-model resumptions caused by them,
   3. input/cached-input processed per resumption.

   Without separating those three, normal agent work and orchestration overhead are difficult to distinguish from account-level usage numbers.
4. jmtt89 commented on Aug 27, 2026

   

   jmtt89

   on Aug 27, 2026

   More actions

   Same pattern on plain CLI, and it removes the two variables your report is most likely to be dismissed on: no Desktop, no subagents, single session.

   Codex CLI 0.149.0, codex exec invoked non-interactively by an orchestrator. One agent, no wait_agent, no roster, no fork/replay. The polling is write_stdin against a unified_exec session. Your per-turn profile reproduces almost exactly:

   	yours	ours
   input/turn	~137–141k	~137k avg
   cached	97.4–98.1%	98.3%
   wait-family share	75% of tool calls	48% of turns (84 of 175)

   Session totals (175 turns): input 23,957,294 · cached 23,542,272 · new input 415,022 · output 52,694 · input:output 454:1.

   Splitting by turn kind isolates the cost:

   	poll turns	work turns
   turns	84	90
   context re-sent	13,507,179	10,450,115
   new input	179,819	235,203
   output	6,151	46,543
   avg output/turn	73	517

   The polls re-metered more context than all the real work combined, for 12% of the output. Cache hit rate on the poll turns alone: 98.7% (13,327,360 of 13,507,179) — so this is your "re-meters full cached context" with the cache working perfectly, and it still cost 13.5M.

   The mechanism, which I think your report is missing and which explains why this is structural rather than a Desktop bug. Checked against 102ae5e2e6ff:

   • core/src/unified_exec/mod.rs:72 — pub(crate) const MAX_YIELD_TIME_MS: u64 = 30_000;
   • core/src/unified_exec/mod.rs:213 — yield_time_ms.clamp(MIN_YIELD_TIME_MS, MAX_YIELD_TIME_MS)
   • core/src/unified_exec/process_manager.rs:575 — clamp_yield_time is applied where the comment reads "For the initial exec_command call"

   So no single tool call can observe a process for more than 30s, including the first one, and it is a compile-time constant with no config override (background_terminal_max_timeout in config_toml.rs:308 bounds how long the process may live, not how long a call may block). Any wait longer than 30s is therefore structurally N re-metered turns, where N = duration/30s. #38495 has the protocol-level framing of the same thing: resumption is pull, not push.

   That matters for triage: your stale-subagent defect is a real amplifier — it manufactures waits that never end — but it is not the cause. Remove Desktop, remove subagents, remove stale rosters, and a single codex exec waiting on an ordinary 21.6-minute CI pipeline still burns 13.5M of metered context to learn nothing.

   Our wait was legitimate, incidentally: the agent had finished, committed, pushed and opened a PR, and was waiting on its own pipeline whose jobs run sequentially with multi-minute gaps. Nothing was hung and it had no signal telling it to stop.

   ---

   Cross-linking, since these threads don't reference each other: #38495 is the mechanism-level report (pull-not-push, no completion push, no timer) and proposes escalating server-side yield + completion push as fixes. It leaves the caching question open — its author suspected a cache miss inflated their numbers. Your data answers it: the account usage API returning ~290M for one day at 97–98% cache is the evidence that cached context is metered regardless. Combined, the two give a conclusion neither has alone: a cache fix would not help, and the yield/push fixes pay for themselves in quota even when caching is perfect.
5. andgedeveloper commented on Sep 13, 2026

   

   andgedeveloper

   on Sep 13, 2026

   More actions

   Also reproducible on Codex CLI 0.154.0 on Linux, with quantified wait/poll token impact

   I can independently reproduce the same class of issue on ChatGPT Pro ($200 / Pro20x) using Codex CLI with a heavily multi-agent software-development workflow.

   Environment

   Codex CLI: 0.154.0
   OS: Arch Linux x86_64
   Install: npm
   Auth: ChatGPT
   Primary model during the measured day: GPT-6 Astra Medium
   Multi-agent, sleep tool, unified exec/code-mode functionality enabled

   I performed a local forensic analysis of one full working day: Sep 12, 2026, America/New_York.

   Day totals from unique response usage records, including compaction:

   8,020 model calls
   912,459,392 cached input tokens
   26,666,608 ordinary input tokens
   2,808,482 output tokens

   The main finding is repeated parent-model re-entry while waiting.

   Polling / wait / status:

   1,574 model calls
   19.63% of all model calls
   198.90M cached input tokens
   21.80% of the day's total cached-input volume

   Of those, 1,474 pure wait/sleep calls alone consumed 186.94M cached tokens.

   This was not primarily caused by unusually huge individual contexts:

   median cached prefix: 114,176 tokens
   p95: 202,368
   p99: 221,952
   max: 242,432
   calls above 300k cached tokens: 0

   So the high token volume is mainly caused by a very large number of model re-entries against an already substantial 100k–200k cached prefix, rather than a few 500k+ context calls.

   One especially clear example:

   A long-running release process was already being waited on inside the shell using a while kill -0 style wait. Despite that process-side wait, the parent/model was re-entered 109 additional times just to continue waiting.

   For that wait loop:

   109 calls
   median interval: ~49.7 seconds
   p90 interval: ~53.6 seconds
   8.49M cached input tokens consumed

   The observed control flow was effectively:

   long-running process/subagent -> tool yield/timeout -> parent model inference -> decide to keep waiting -> wait again -> timeout/yield -> parent inference -> repeat

   Two large working threads demonstrate the same behavior:

   Thread A

   1,364 total model calls
   332 polling/status calls
   43.31M cached tokens from polling/status

   Thread B

   580 total model calls
   217 polling/status calls
   25.02M cached tokens from polling/status

   This workflow does use many engineering/architecture reviewers and subagents, but the reviewers themselves do not appear to inherit huge parent contexts.

   I checked 22 engineering-reviewer threads: their initial input was only about 20.3k–20.9k tokens, with separate thread IDs and no previous parent tool outputs. The expensive behavior is primarily the parent/model repeatedly waking while waiting for those independent agents/processes.

   There also appears to be an interaction with progress/heartbeat behavior: the environment encourages the agent not to block waiting for long periods and to provide periodic progress updates. In practice, this can couple a user-visible heartbeat to another full model inference even though no new reasoning decision is required.

   Expected behavior

   I would expect long-running waits to be runtime/event driven:

   while a child/process is still running, keep waiting below the model layer;
   child completion/failure should immediately wake the parent;
   user steering/cancellation should immediately wake the parent;
   an actual decision-required event should wake the parent;
   progress/heartbeat UI should ideally not require a full parent-model inference;
   timeout-only wait events should not repeatedly re-meter the complete parent context when nothing changed.

   In other words, ideally:

   parent -> start/wait for child -> runtime waits -> child completes -> wake parent model

   rather than:

   parent -> wait -> timeout -> full parent inference -> wait -> timeout -> full parent inference -> ...

   I also verified that this is not a ccusage cumulative-counter double-counting artifact. Duplicate unchanged token-count records were excluded, unique response IDs reproduce the usage totals, and including otherwise omitted compaction requests actually increases the measured total slightly.

   For context, this workload consumed approximately 80% of my Pro20x weekly allowance during that working day. I am not claiming that polling caused all 80%; the specific measurable finding is that wait/poll/status re-entry alone represented ~21.8% of cached-input volume.

   This looks closely related to #35108, #35259 and the wait_agent timeout / parent polling behavior discussed in #41875.

   I can provide additional sanitized per-call metrics, thread timelines, compaction data, or wait-loop aggregates if they would be useful to the Codex team.