Codex CLI repeatedly wakes xhigh to poll deterministic long-running jobs, exhausting finite weekly usage before task completion #45974

Activity

1. jmcclure8275

   added

   bugSomething isn't workingSomething isn't working

   on Sep 16, 2026
2. github-actions

   added

   CLIIssues related to the Codex CLIIssues related to the Codex CLI

   model-behaviorIssues related to behaviors exhibited by the modelIssues related to behaviors exhibited by the model

   rate-limitsIssues related to rate limits, quotas, and token usage reportingIssues related to rate limits, quotas, and token usage reporting

   on Sep 16, 2026
3. github-actions commented on Sep 16, 2026

   

   github-actionsbot

   on Sep 16, 2026 – with GitHub Actions

   Contributor

   More actions

   Potential duplicates detected. Please review them and close your issue if it is a duplicate.

   • measured cost of [false] goal continuations across 3,808 sessions #44909
   • Feature: Claude Code-style monitor tool with first-class remote-host support #44855

   Powered by Codex Action
4. jmcclure8275 commented on Sep 16, 2026

   

   jmcclure8275

   on Sep 16, 2026

   Author

   More actions

   Additional timing/telemetry detail requested by OpenAI Support:

   Affected uploaded thread: 01a098e5-0707-7530-9138-688b82a1e86e

   This was one continuous Codex CLI session running:

   model: gpt-6-astra
   reasoning_effort: xhigh

   The session hit usage_limit_exceeded three times as I applied banked resets:

   2026-09-14 20:38:20 CDT
   2026-09-15 13:04:28 CDT
   2026-09-16 06:54:10 CDT

   Equivalent UTC timestamps from the rollout:

   2026-09-15T01:38:20.857Z
   2026-09-15T18:04:28.963Z
   2026-09-16T11:54:10.143Z

   All of the additional capacity I applied during this run came from banked resets, not purchased instant resets and not normal weekly-window resets.

   One diagnostic detail that may be important: although the active model was gpt-6-astra at reasoning_effort: xhigh, nearby token_count telemetry identifies:

   limit_id: codex_bengalfox
   limit_name: GPT-5.3-Codex-Spark

   with that separate meter showing used_percent: 0.0.

   OpenAI Support has clarified that codex_bengalfox / Codex Spark is a separate meter from the primary Codex weekly limit, so I am not claiming that the 0.0 → usage_limit_exceeded sequence by itself proves incorrect usage accounting.

   I am including the mismatch because Support specifically identified it as useful diagnostic information: the session performing the repeated polling was gpt-6-astra / xhigh, while this telemetry line references the separate codex_bengalfox meter.

   The underlying engineering issue remains the model-mediated polling behavior described in the issue: xhigh continued waking to supervise healthy deterministic jobs until the primary weekly limit was exhausted.
5. lucaschang2021 commented on Sep 17, 2026

   

   lucaschang2021

   on Sep 17, 2026 – with ChatGPT Codex Connector

   More actions

   I dug into current main and the related reports (#44909, #32188). I think #45974 is best fixed at the runtime ownership boundary, not primarily with prompt tweaks or shorter polling intervals.

   What current main appears to do

   There are two pieces that line up with the observed behavior:

   1. Goal continuation wakes inference as soon as the thread goes idle.
      GoalExtension::on_thread_idle calls GoalRuntimeHandle::continue_if_idle(), and continue_if_idle() eventually calls start_turn_if_idle(...) for an active goal. There is a continuation-deferral check, but it is not currently tied to a live deterministic process/subagent wait.

   2. Unified exec already knows when a background process exits, but that completion does not become an idle-thread wake.
      unified_exec::async_watcher::spawn_exit_watcher waits for process completion and emits ExecCommandEnd. That is the seam described in Event-driven wakeup when background exec sessions complete #32188: the machine can wait cheaply, but the idle agent is not resumed from that event.

   This also matches the maintainer note in #44909: the newer empty-response circuit breaker helps empty automatic turns, but repeated polling/tool calls still keep an active goal alive and reset the breaker.

   Proposed behavior

   For deterministic machine-owned work, I would make the lifecycle explicitly event-driven:

   model reasons
     -> launches deterministic work
     -> runtime registers a machine-wait handle
     -> goal continuation is deferred
     -> model turn ends
     -> process/subagent runs with zero model inference
     -> completion/failure event arrives
     -> runtime enqueues one bounded completion event
     -> deferral is cleared
     -> exactly one continuation turn starts
   

   In other words: machine progress should satisfy goal liveness without requiring model progress.

   Implementation shape

   I would reuse/converge with #32188 rather than build a separate polling-specific mechanism:

   • Add a race-safe completion registration for yielded exec_command / subagent work (armed, completed, observed is a good state model).
   • When an active goal yields a known-live deterministic handle, persist/set goal-continuation deferral instead of immediately starting another goal turn.
   • Let the existing process/subagent watcher own the wait.
   • On normal completion/failure, enqueue a bounded completion fragment and schedule exactly one continuation.
   • If the completion is manually observed (write_stdin / explicit wait) first, mark it observed and suppress the later wake.
   • A kill/interrupt/session teardown must not create a phantom wake.
   • If the thread/session is at a concurrency/capacity ceiling, queue or degrade to current behavior rather than oversubscribing.
   • Multiple near-simultaneous completions should be batched/coalesced where possible.

   Fallback for waits the runtime cannot classify

   For cases where Codex still ends up in an unclassified poll-only loop, an exponential backoff/circuit breaker is still useful, but I would treat it as a fallback rather than the primary solution. The primary invariant should be: if the runtime already owns a live process/subagent handle, inference should not be used as the clock.

   Regression tests I would want

   1. Active goal starts a background process, turn becomes idle, process runs for N seconds: zero model continuations during N and exactly one wake on exit.
   2. Process exits before the wake registration is fully armed: completion is not lost.
   3. Manual write_stdin observes completion first: no duplicate automatic wake.
   4. Process is killed/aborted or session is torn down: no wake.
   5. Completion lands at the final-answer/idle boundary: exactly once.
   6. Several processes finish together: bounded/coalesced continuation behavior.
   7. Goal remains active across the machine wait and resumes with the completion result, rather than being paused/blocked just to avoid token burn.

   This would address the #45974 failure mode without sacrificing the intended “gritty” goal semantics: the goal stays alive, but expensive intelligence sleeps while cheap deterministic execution is making progress.
6. lucaschang2021 commented on Sep 17, 2026

   

   lucaschang2021

   on Sep 17, 2026 – with ChatGPT Codex Connector

   More actions

   Follow-up after checking the latest #32188 discussion and current main:

   There is now stronger evidence that this is an idle-wakeup / wait-ownership problem rather than a model-prompt problem.

   New evidence

   1. Event-driven completion has been implemented successfully in personal forks.
      Event-driven wakeup when background exec sessions complete #32188 now includes a tested implementation where a yielded background command can finish after the assistant turn ends, enqueue a bounded completion notification, and resume the same thread exactly once — with no write_stdin polling loop required.

   2. codex queue --thread ... --message ... already proves the thread can be woken from an external completion event.
      That workaround is not sufficient as a product solution (it lacks native cancellation/dedup/capacity semantics), but it confirms the missing primitive is not “can Codex ever wake an idle thread?”; it is “can the runtime wire owned background work into that wake path safely?”

   3. A useful benchmark proposed in Event-driven wakeup when background exec sessions complete #32188 is:

      WAIT_MAIN_INPUT = total parent-model input consumed exclusively while waiting for external work to reach a terminal state

      For a runtime-managed wait, the target should be 0 until an actual completion/failure event exists.

   Minimal invariant I think Codex should enforce

   If the runtime owns a live deterministic handle, the model should not be used as the clock.

   owned live handle exists
     => defer automatic goal continuation
     => no model wake just to observe liveness
     => watcher owns waiting
     => terminal event enqueues bounded completion state
     => clear deferral
     => schedule exactly one continuation
   

   The important point is that this should apply to both:

   • yielded unified_exec processes; and
   • subagent/task handles where completion is already machine-observable.

   Otherwise fixing only shell-process waits will still leave the same quota burn through wait_agent / task-status polling.

   Suggested integration point with Goals

   Current main already has thread_goal_continuation_deferrals. I think that mechanism can be made semantic rather than incidental:

   • set/persist deferral when the goal is active and there is at least one known-live runtime-owned wait handle;
   • do not classify the goal as paused/blocked merely because it is waiting;
   • clear deferral only when a relevant completion/failure/cancellation transition is committed;
   • then allow start_turn_if_idle(...) to run once with the completion payload.

   That preserves the intended gritty behavior while removing inference-only liveness checks.

   Additional regression criteria

   Beyond the earlier list, I would add:

   • subagent parity: background subagent completion must satisfy the same zero-poll / exactly-once wake semantics as background process completion;
   • wait-cost metric: WAIT_MAIN_INPUT == 0 during pure machine wait;
   • durability boundary is explicit: if wake delivery is not durable across app/CLI restart, make that behavior explicit rather than silently losing responsibility;
   • no wake amplification: several completions close together should produce one bounded continuation batch where possible;
   • no double-observation: an explicit/manual consume path must invalidate the pending automatic wake atomically.

   I think this is the cleanest way to address #45974, #44909, and #32188 as one runtime scheduling problem: model reasoning should resume on new information, not on elapsed time alone.
7. lucaschang2021 commented on Sep 17, 2026

   

   lucaschang2021

   on Sep 17, 2026 – with ChatGPT Codex Connector

   More actions

   Implementation refinement after checking current main more closely (around 8452164c): I would not use the existing thread_goal_continuation_deferrals row as the machine-wait registry itself.

   The current schema is one row per thread:

   CREATE TABLE thread_goal_continuation_deferrals (
       thread_id TEXT PRIMARY KEY NOT NULL REFERENCES thread_goals(thread_id) ON DELETE CASCADE
   );

   That works as a thread-level boolean, but it cannot represent multiple concurrent waits. If a goal owns jobs A/B/C and A completes first, clearing a single boolean deferral at A's completion can re-enable automatic goal continuation while B and C are still running, recreating the same polling burn.

   A more robust shape

   I think the runtime needs a small keyed wait registry, e.g. conceptually:

   MachineWait {
     owner_thread_id,
     goal_id?,
     kind: Exec | SubAgent,
     handle_id,
     state: Armed | Completed | Observed | Cancelled,
     wake_policy,
   }
   

   Then goal deferral becomes a derived invariant:

   defer goal continuation
       iff EXISTS live runtime-owned wait for this goal/thread
   

   rather than a boolean that one completion can clear prematurely.

   Subagents are already very close to this

   Current AgentControl::maybe_start_completion_watcher already does the expensive part correctly: it creates a detached watcher, subscribes to the child status, and waits until the child reaches a final state.

   For MA v2, the watcher then builds the completion message with:

   InterAgentCommunication::new(..., /*trigger_turn*/ false)

   and for the legacy path it calls:

   inject_fragment_without_turn(...)

   So Codex already has machine-observable subagent completion; the completion is deliberately delivered as non-waking state.

   Also, send_inter_agent_communication already has a trigger_turn path and performs the execution-capacity check before starting a turn. That means the subagent fix does not need another polling layer or another watcher. It needs an ownership rule that conditionally converts a completion from queue-only delivery into an exactly-once wake when the parent has an armed runtime wait for that child.

   Suggested lifecycle

   1. exec/subagent yields a live handle
   2. register MachineWait before exposing the yielded state
   3. active goal sees live wait => continuation remains deferred
   4. parent model turn ends
   5. watcher waits with zero inference
   6. terminal event atomically transitions that wait
   7. enqueue bounded completion payload
   8. if parent is idle, schedule one turn; if active, retain/queue
   9. explicit/manual observation marks the wait Observed and suppresses a later wake
   10. cancellation/kill/teardown removes the wait without a wake
   11. remaining live waits keep the goal deferred
   

   The ordering in steps 6–8 matters: enqueue/commit the new information before reopening continuation, otherwise there is a race where the goal wakes and still has nothing new to reason about.

   Multi-wait regression I would add

   Parent goal launches A, B, C concurrently:

   • while all three are running: WAIT_MAIN_INPUT == 0;
   • A completes: exactly one parent wake with A's completion;
   • after that turn, B and C still count as live machine waits, so the goal returns to idle without polling;
   • B/C complete later and wake the parent independently or coalesce if their completions race;
   • manual wait_agent / write_stdin observation of any handle suppresses only that handle's pending automatic wake.

   This is the main correction I would make to my previous suggestion: reuse the existing goal-deferral gate, but back it with keyed wait ownership/multiplicity rather than treating the current thread-level boolean as the wait registry.

   That would make process and subagent waits share the same scheduling invariant while preserving current capacity controls and the intended goal semantics.
8. ShaneBreazeale commented on Sep 20, 2026

   

   ShaneBreazeale

   on Sep 20, 2026

   Last edited by ShaneBreazeale

   More actions

   Additional evidence and a working interim pattern for the event-driven fix proposed here:

   A recent independent report measured the polling overhead in two runs:

   • 19 short sleep/check calls accounted for 44% of the orchestrator's estimated cost.
   • With Astra High orchestrating Luna Max, the polling version made 41 orchestrator responses and cost about $1.20 during the worker phase. The completion-driven version needed one response and cost about $0.04. Both passed the same 244 tests.

   Source: https://www.reddit.com/r/codex/comments/1wlcy5q/comment/paxyixk/

   I also validated the interim workaround on Codex CLI 0.155.1. Tool processes receive CODEX_THREAD_ID and CODEX_SESSION_ID, so a detached launcher can capture the current thread, run a non-interactive command with output/status persisted to files, and invoke exactly once on exit:

   codex queue --thread "$CODEX_THREAD_ID" --message "Job finished; inspect <status> and <output>, verify, and continue."
   

   The originating model turn can end immediately. The queue message wakes the same thread after success, failure, or launch error, with no model calls while the command is merely alive.

   One integration wrinkle is also reproducible: when the detached wrapper inherits a managed workspace sandbox, the later codex queue call can fail opening $CODEX_HOME/state_5.sqlite with attempt to write a readonly database. Launching the wrapper with narrowly scoped escalated access works, but adds an approval/configuration burden. A native runtime completion wake would avoid requiring a background helper to mutate Codex state from inside the command sandbox.

   This makes the desired product behavior concrete, but it should be a native runtime contract rather than a user-maintained wrapper:

   1. A yielded exec/subagent handle registers on_exit: wake (or an equivalent completion policy).
   2. The runtime owns the wait and goal continuation remains deferred while relevant handles are live.
   3. Completion commits a bounded result before scheduling one continuation.
   4. Manual observation atomically suppresses the pending wake; cancellation and teardown do not create one.
   5. Concurrent completions coalesce where practical.

   The regression metric suggested above captures the intended invariant well: WAIT_MAIN_INPUT == 0 until a terminal event introduces new information. codex queue proves the wake path already exists; the missing part is wiring runtime-owned completion to it automatically and steering the model away from elapsed-time polling.
9. PHPCraftdream

   added a commit that references this issue on Oct 1, 2026

   docs: explain temporary cli-run workaround (openai/codex#42908, opena…

   ...

   1118642
10. PHPCraftdream commented on Oct 1, 2026

    

    PHPCraftdream

    on Oct 1, 2026

    More actions

    @lucaschang2021's distinction between a machine-owned wait and goal continuation, plus @ShaneBreazeale's queue/sandbox example, helps separate two problems here. A command runner can remove the need to ask xhigh whether a healthy job is still alive; it cannot by itself prevent an active goal from scheduling unrelated model turns. I have packaged a temporary workaround for the first part, not a fix for the whole goal-governance issue.

    I maintain cc-arch-hands, whose cli-run MCP integration is available in 0.15.1. A run call with queue delivery hands a named set of commands to a detached Node worker and acknowledges its UID. The worker owns process waiting, captures stdout/stderr and exit results on disk, and attempts one short codex queue completion message to the calling thread after all jobs finish. The caller can end its turn rather than repeatedly issuing empty write_stdin calls or rereading progress JSON. Later, status and logs allow inspection by task name or UID; they are not intended as a new polling loop.

    npx cc-arch-hands@0.15.1 install --codex-skills

    That installs the optional Codex skills, local stdio MCP registration, and marked instructions for routing long-running commands. Restart Codex to load them. Implementation and the coverage matrix.

    The permission wrinkle in the previous comment is real and should not be hidden by an installation command. MCP workers do not inherit Codex's per-turn filesystem sandbox; run can execute arbitrary commands and therefore requires approval or an explicit user trust decision. We recently encountered Codex rejecting the MCP call before launch with MCP tool call requires approval, but approval policy is never. Installation does not silently override that policy. Version 0.15.1 preserves an explicitly selected tools.run.approval_mode during reinstall. A missing/unloaded target thread or queue-access error can still prevent notification; that failure is saved with the run state.

    This runner does not implement thread_goal_continuation_deferrals, register native subagent wait handles, coalesce independent task completions at the scheduler level, or guarantee WAIT_MAIN_INPUT == 0 for a session with an active goal. We also have no controlled token-cost benchmark, so the savings reported earlier in this thread are not measurements of our tool.

    It is usable now for moving deterministic CLI waiting out of the model loop, while the upstream ownership invariant remains necessary: live machine work should defer inference, and terminal information should be committed before reopening goal continuation. I would keep this issue open for that runtime-level change rather than treat an external queue helper as its resolution.

    👍React with 👍1Reacted by Lucaschang
11. lucaschang2021 commented on Oct 1, 2026

    

    lucaschang2021

    on Oct 1, 2026 – with ChatGPT Codex Connector

    More actions

    Thank you for building this and, especially, for drawing the boundary so clearly.

    I think your runner is a very useful validation of the distinction I was trying to make: deterministic machine-owned waiting can be moved out of the model loop today, while goal continuation still needs a runtime-level ownership rule.

    The fact that you explicitly do not treat the external queue helper as a full resolution is important. The remaining invariant is still upstream: while live machine work owns progress, continuation inference should be deferred, and terminal state should be committed before the goal is reopened.

    Also appreciate you documenting the approval/sandbox limitation instead of hiding it behind the installer. That makes the workaround much easier to reason about as an engineering component rather than as a substitute for native scheduling semantics.

    Thanks again for implementing and testing this in practice — it gives the machine-wait half of the proposal a concrete reference point.

    👍React with 👍1Reacted by Shane