Skip to content

MultiAgentV2: queue-only message to a completed agent pins a residency slot #32353

Open

Description

@zhibisora

What version of Codex CLI is running?

Observed with codex-cli 0.144.0 bundled in Conductor. I also checked codex-cli 0.144.1 and current main (5c19155cbd93bfa099016e7487259f61669823ff); the relevant v2 residency and message-delivery logic is unchanged.

What platform is your computer?

Darwin 25.5.0 arm64 arm

Surface: Codex app-server / MultiAgentV2, accessed through Conductor 0.74.0.

What issue are you seeing?

A completed MultiAgentV2 subagent can indefinitely occupy a resident thread slot when it has a pending queue-only mailbox message. Once the other resident slots are active, a new spawn_agent fails even though the number of actively running subagents is below the configured limit:

collab spawn failed: agent thread limit reached

This is not cumulative thread exhaustion. In the observed session, spawning succeeded again later after the pinned mailbox was consumed. The same AgentLimitReached error currently represents active concurrency exhaustion and residency exhaustion, so the parent cannot tell which condition occurred or which agent is blocking eviction.

What steps can reproduce the bug?

Use a small v2 capacity so the behavior is visible:

[features.multi_agent_v2]
max_concurrent_threads_per_session = 4

The value includes the root agent, leaving three subagent resident slots.

  1. Spawn subagent A and let it reach Completed.
  2. Call send_message targeting A. In v2 this is queue-only and does not trigger a new turn.
  3. Spawn subagents B and C with tasks that remain active.
  4. Attempt to spawn subagent D.
  5. D fails with collab spawn failed: agent thread limit reached, although only B and C are active.
  6. Call followup_task on A so it wakes, consumes its mailbox, and completes again. Retry the spawn; it can now succeed.

The production occurrence was the race-shaped variant: the root called send_message near A's completion boundary. A completed about one second later, leaving the queue-only message pending. With B and C active, spawning D failed.

What is the expected behavior?

A queue-only message should not leave a completed agent as a hidden, indefinitely non-evictable resident.

Any of these behaviors would avoid the deadlock:

  • reject send_message to an already completed agent and tell the caller to use followup_task;
  • automatically trigger a follow-up turn when a queue-only message crosses the target's completion boundary;
  • persist the mailbox independently so the completed thread can be unloaded and resumed later;
  • expose an explicit close/drain operation for completed v2 agents.

At minimum, AgentLimitReached should distinguish active concurrency from residency pressure and report data such as max, residents, active, and pending_mailbox_blockers with an actionable recovery hint.

Additional information

Current main makes the interaction explicit:

Related but distinct:

Increasing features.multi_agent_v2.max_concurrent_threads_per_session reduces the frequency but does not fix the pinned-mailbox lifecycle bug.

Activity

  1. added
    bugSomething isn't working
    app-serverIssues involving app server protocol or interfaces
    subagentIssues involving subagents or multi-agent features
    on Jul 11, 2026
  2. Yurdos-r commented on Jul 21, 2026

    @Yurdos-r

    Independent reproduction from Codex Desktop (2026-07-21)

    I reproduced the admission failure with a controlled baseline and a pending-mailbox variant.

    Environment

    • Surface: Codex Desktop on Windows
    • App/CLI version: not exposed to the isolated test thread, so I cannot report it reliably
    • Observed capacity: root plus three direct subagent resident slots
    • No files or configuration were changed

    Baseline: ordinary completed agents were reclaimable

    1. In a fresh isolated task, spawn direct subagents A, B, and C with clean history.
    2. Let all three return A_OK, B_OK, and C_OK and reach completed.
    3. Spawn a new direct subagent D.

    Result: D's spawn_agent returned successfully without a noticeable wait, and D completed with D_OK.

    This indicates that ordinary completed agents alone did not block a later spawn in this runtime.

    Variant: a queue-only message pinned the completed resident

    1. In a second fresh isolated task, spawn A and let it complete with A_OK.
    2. Send STATUS_PROBE to completed A using queue-only send_message. It did not trigger a new turn.
    3. Spawn E and F with tasks that remain active for about 45 seconds.
    4. While E and F were active, attempt to spawn G.

    The G spawn failed immediately with the exact output:

    collab spawn failed: agent thread limit reached
    

    At that moment only E and F were actively running. A was completed, but it had the pending queue-only mailbox item. The controlled difference from the successful baseline was the pending mailbox on A.

    I then called followup_task on A; A consumed the queued work and returned A_DRAINED. E and F also completed. A subsequent retry could not be emitted because the isolated test task encountered a separate tool-routing failure, so I am not claiming that the recovery path was independently verified here.

    What this reproduction supports

    • Normal completed residents were evictable/reclaimable in the baseline.
    • A completed resident with pending queue-only mail continued to consume residency capacity.
    • With that pinned resident plus two active children, a fourth direct child was rejected using the same generic AgentLimitReached error as true active-concurrency exhaustion.
    • The current error gives no indication that a completed agent's mailbox is the blocker.

    This independently supports the pending-mailbox residency mechanism described in the issue and the need for an actionable error that distinguishes active concurrency from residency pressure.

  3. KNaiFen commented on Aug 5, 2026

    @KNaiFen

    I audited this against current main at c38a60ded2ffbad510f8b512e2902f0128e998e8. The reported residency mechanism is still present, and there is an important race constraint for a fix.

    Why a tool-handler status precheck is insufficient

    handle_message_string_tool resolves and loads the receiver, then submits an Op::InterAgentCommunication. The actual mailbox insertion happens later in session::handlers::inter_agent_communication:

    message_tool.rs -> AgentControl::send_inter_agent_communication
                    -> ThreadManagerState::send_op
                    -> session::handlers::inter_agent_communication
                    -> InputQueue::enqueue_mailbox_communication
    

    A Completed check in message_tool.rs alone can pass while the target is still running, followed immediately by target completion before the queued op reaches enqueue_mailbox_communication. That recreates the same completed-plus-pending-mailbox blocker.

    Current task completion calls maybe_start_turn_for_pending_work, but that helper intentionally starts only for trigger_turn mail (or durable sleep). Queue-only mail therefore remains pending, and residency::is_unloadable rejects the resident because has_pending_mailbox_items() is true.

    Invariant a fix should enforce

    After the target reaches a terminal state, queue-only mail must not become an indefinite residency blocker. The atomic decision needs to live at the actual mailbox admission/completion boundary, not only in the model-facing tool handler.

    Two viable semantics appear to require maintainer direction:

    • Promote queue-only mail that crosses the running-to-terminal boundary into a wake-up turn, preserving the message and draining the resident.
    • Reject terminal queue-only admission and return an actionable use followup_task error. This needs an acknowledgement path from the session handler back to the tool call; the current op submission only confirms that the op was accepted by the session loop.

    Regression coverage

    A focused test should cover both orderings:

    1. Target A is already completed before send_message(A).
    2. A completes after the tool-side check but before mailbox admission.
    3. With A plus two active residents at capacity 3, spawning D must either succeed after the defined drain behavior or return the explicit delivery error, never generic AgentLimitReached caused by a hidden queue-only blocker.
    4. Assert that A has no stranded queue-only mailbox item after the operation settles.

    Per the invitation-only contribution policy, I will not open an unsolicited PR. If a maintainer selects the intended delivery semantic and invites a contribution, I can implement the narrow fix and integration test.

  4. wuyak commented on Aug 9, 2026

    @wuyak

    This issue also exposes a message-lifecycle boundary that is currently difficult for the caller to observe:

    accepted by the session loop
      != admitted to the mailbox
      != consumed by a child turn
      != applied as steering
      != safe to discard
      != resident thread reclaimable
    

    I opened #37744 for the complementary protocol gap: a parent should be able to distinguish queued, consumed, applied, rejected, superseded, and unknown, rather than treating a completed send_message call as proof of application.

    At a terminal boundary, the runtime should define one deterministic behavior: reject queue-only delivery and direct the caller to a follow-up turn, promote the message into a new turn, or externalize the pending mailbox so it does not pin residency. Whichever behavior is selected, the resulting delivery and reclaim state should be observable.

  5. theosp commented on Aug 25, 2026

    @theosp

    Corroborating production reproduction

    Observed on Codex Desktop 26.818.61809 with bundled CLI 0.149.0-alpha.4.3 in session 01a03411-e72a-73a2-aae2-97872a621dab.

    The parent called queue-only send_message on child A while A was running; A completed shortly afterward. Two fresh siblings then started, but a third fresh spawn failed with agent thread limit reached. At failure, the list showed the root plus the two siblings running and A still completed. Calling interrupt_agent on A reported its prior completed status but did not free admission, and the third spawn failed again.

    This closely matches the race-shaped reproduction described here: a message crossing the completion boundary appears to leave the completed child resident and non-evictable. It also explains the broader symptom reported in #39694. The session had previously created 16 fresh children over multiple waves, but the evidence supports residency pressure—not a cumulative 16-child limit.

  6. lucasgnz commented on Sep 9, 2026

    @lucasgnz

    Additional reproduction and local prototype validation on 0.153.4 (macOS arm64)

    A Codex-assisted investigation reproduced this specific pending-mailbox case against the official rust-v0.153.4 sources (3d2ee51), using the existing mocked Responses integration harness rather than production conversations.

    Minimal integration scenario:

    1. Set V2 capacity to 2 (root plus one child).
    2. Spawn worker A and wait for TurnComplete.
    3. Send queue-only mail to A with send_message.
    4. Spawn replacement worker B.
    5. Resume A with followup_task and verify its original identity, inherited permissions/tools, and queued message are preserved.

    Step 4 fails on the unmodified tag: A cannot be evicted because is_unloadable requires an empty mailbox. The new integration test fails on the original source, including the automatic retry.

    A local prototype makes the scenario pass by retaining an Arc<InputQueue> in the session-scoped agent registry alongside evicted environment state. A reloaded session reuses that queue; cached eviction state is cleared only after successful reload. Terminal, idle residents with queue-only mail become evictable, while trigger-turn mail and durable-sleep sessions remain protected. This retains the queue within the same application process; it does not add crash-durable mailbox storage.

    The same eviction path also needs reservation/cancellation coverage: a candidate temporarily removed from the LRU during awaited shutdown must not appear as free capacity to another reservation, and cancellation must restore or finish reclaiming that candidate. The prototype serializes residency reservations and restores candidates on cancelled eviction, including retry after shutdown.

    Validation with Rust 1.97.1:

    • 95/95 focused agent-control, registry and input-queue tests passed, including completed/errored/interrupted residents, preserved mail, pending follow-ups, concurrent reservations and cancelled eviction.
    • 7/7 execution/resume integration tests passed. The cold-resume fixture needed to clean up only still-loaded workers, since the sibling spawn can now legitimately evict a completed worker; its identity/role/permission assertions remain intact.
    • Full codex-core run: 4,081 passed, 6 failed, 24 skipped. All six remaining failures were independently reproduced on the unmodified tag: four missing auxiliary-binary failures, one code-mode elicitation test and one sandbox-environment expectation.

    As checked on September 8, the residency and registry files are unchanged in rust-v0.154.0-alpha.8 and main at 5ac0b87. The relevant queue-only admission condition also remains present.

    Sharing this as reproduction and design evidence under the contribution policy. The prototype has not been deployed into the Desktop runtime or validated across platforms. No production logs, private paths, task identifiers, or project content are included.

  7. 2217173240 commented on Sep 23, 2026

    @2217173240

    Additional production trace on bundled CLI 0.155.0-alpha.16 (macOS arm64)

    This is corroborating trace evidence for the completion-boundary variant, not a claim of a new deterministic reproduction or a confirmed in-memory mailbox inspection. Agent names below are anonymized.

    Environment:

    • Codex desktop app 26.917.51856 (build 10492)
    • Bundled CLI 0.155.0-alpha.16
    • macOS 26.3, arm64
    • MultiAgentV2; the affected runtime announced seven total concurrency slots including the root.

    Observed sequence on 2026-09-23 (UTC):

    Timestamp Event
    05:54:33.709 Parent invokes queue-only send_message to worker A.
    05:54:34.027 A records task_complete.
    05:54:34.618 The parent's send_message returns successfully.
    05:56:27.008 A replacement spawn_agent fails with collab spawn failed: agent thread limit reached.
    05:56:27 onward Repeated lists retain A as completed, while ordinary completed siblings disappear and are replaced.
    07:13:06.238 List contains the root, five running siblings, and A still completed.
    07:21:08.227 A remains completed in the resident list, about 86 minutes after its task completed.

    No subsequent followup_task or interrupt_agent targeting A appears in the inspected interval. The child rollout contains no additional turn after the recorded completion. This is consistent with a stranded queue-only message, although the private in-memory mailbox and residency permit state were not inspected, so the causal mechanism remains inferred from the trace and source.

    There was a separate local configuration rollback during this broader investigation. I excluded that from the lifecycle claim above: the affected runtime already announced seven slots throughout this interval. This report does not claim a model-specific concurrency limit or that changing the limit fixes the lifecycle issue.

    Source check against main at 40eac3ce8a0c10cbcb9db910d529355eb2f8fc09:

    • is_unloadable still requires !has_pending_mailbox_items() for a terminal idle worker.
    • Mailbox admission enqueues the communication and starts pending work only for trigger-turn mail or outstanding durable sleep.
    • The published rust-v0.155.0-alpha.16 source has the same pending-mailbox unloadability restriction.

    Related merged PR #47240 protects queued submissions and makes eviction/removal race-safe. It should not be assumed to resolve this terminal queue-only mailbox case; the restriction above remains on main. I have not run a reproduction against a main build.

    A useful regression case remains the already-described controlled completion/admission interleaving, checking that messages are preserved and that settled terminal workers do not indefinitely block replacement admission. An observable distinction between active capacity and non-evictable mailbox residency would also help operators diagnose this without interrupting healthy siblings.

    No customer content, credentials, local paths, raw rollouts, or private task identifiers are included. This report was prepared with Codex assistance from local trace inspection; the live failing task was not restarted or forcibly terminated for this report.

  8. chunxiaoyi commented on Sep 26, 2026

    @chunxiaoyi

    MultiAgentV2: queue-only send_message stranded in a subagent's mailbox pins its residency slot for the rest of the session → agent thread limit reached in a strictly serial workflow

    Related: #32353 (same root cause, idle-target variant), #39694 (same symptom)

    Summary

    In a strictly serial orchestration (never more than one subagent running at a time), spawn_agent started failing with collab spawn failed: agent thread limit reached after 8 successful spawns. At the time of failure, no subagent was running.

    The 3 resident slots were held by the only 3 subagents in the session that had ever received a send_message from the root. In each case at least one of those queue-only messages was never drained by the recipient: the recipient either finished its turn a few seconds after the message arrived, or was interrupted while the message was still queued. A subagent whose mailbox is not empty never passes is_unloadable, so it can never be evicted. After the third such subagent, every later spawn failed. The other 5 subagents never received a message, and all of them were evicted normally.

    This extends #32353. The target does not have to be idle when the message is sent. A message sent to a running subagent can also get stuck, for either of two reasons:

    1. Interrupted before reading it. The subagent is interrupted before it makes another model call, so it never reads the message. Interrupting does not clear the mailbox.
    2. Final answer already in progress. The subagent has already started writing its final answer. The message is deferred to a "next turn" that never starts.

    Environment

    Codex CLI 0.157.1 (Linux, TUI)
    Orchestrator model gpt-6-sol (catalog reports multi_agent_version = v2, tool_mode = code_mode_only)
    Multi-agent config defaults: max_concurrent_threads_per_session = 4 → 3 subagent slots; no overrides in config.toml
    Orchestration pattern Root spawns one fresh subagent per stage, waits with wait_agent, collects output, then spawns the next. Never more than one subagent running.

    Timeline

    2026-09-26, times in UTC+08:00. Subagents are renamed W1–W9 in spawn order.

    Evidence used:

    • Tool calls and results: from the root rollout.
    • Turn start/end and consumed messages: from each subagent's own rollout. A consumed queue-only message leaves an inter_agent_communication_metadata record with trigger_turn: false; an unconsumed one leaves nothing.
    • Eviction order: Codex does not log it. It is reconstructed by replaying the LRU policy in agent/control/residency.rs. The replay matches all 8 successful and 3 failed spawns.
    Time Event Resident set after the event (LRU → MRU)
    18:10:42 Root session starts —
    18:13:34 spawn W1. It completes at 18:14:40 W1
    18:15:15 spawn W2 (running) W1, W2
    18:17:34 root send_message → W2 (W2 running). Consumed by W2 at 18:17:39 W1, W2
    18:17:48 W2 send_message → root
    18:17:53 root send_message → W2 (W2 still running) W1, W2
    18:17:56 W2 turn completes, 3 s after the second message. No consumption record for the 18:17:53 message. W2 is now pinned W1, W2
    18:19:07 spawn W3. It completes at 18:20:19 W1, W2, W3
    18:20:59 spawn W4: evicts W1. W4 completes at 18:22:10 W2, W3, W4
    18:22:44 spawn W5: skips W2 (pending mail), evicts W3. W5 completes at 18:25:25 W2, W4, W5
    18:26:16 spawn W6: skips W2, evicts W4. W6 completes at 18:27:12 W2, W5, W6
    18:27:57 spawn W7: skips W2, evicts W5. W7 is running W2, W6, W7
    18:30:24 root send_message → W7 (running). Never consumed
    18:35:56 root send_message → W7 (running). Never consumed
    18:37:07 root interrupt_agent → W7. W7 turn aborted (interrupted) with 2 messages still queued. W7 is now pinned W2, W6, W7
    18:37:25 spawn W8: skips W2 and W7, evicts W6. W8 is running W2, W7, W8
    18:39:41 root send_message → W8 (running). Never consumed
    18:40:59 root send_message → W8 (running). Never consumed
    18:43:19 root interrupt_agent → W8. W8 turn aborted (interrupted) with 2 messages still queued. W8 is now pinned W2, W7, W8
    18:44:04 spawn W9 → collab spawn failed: agent thread limit reached W2, W7, W8
    18:44:10 list_agents: no subagent running
    18:45:08 spawn W9 (retry) → agent thread limit reached
    18:45:45 spawn W9 (retry) → agent thread limit reached. Logged at ERROR by codex_core::tools::router

    State of the resident set at 18:44:04:

    Subagent Status Unread queue-only messages is_unloadable
    W2 Completed 1 (18:17:53) false
    W7 Interrupted 2 (18:30:24, 18:35:56) false
    W8 Interrupted 2 (18:39:41, 18:40:59) false

    Running subagents: 0. The running-turn limit was not reached; the residency limit was.

    Across the whole session, W2, W7 and W8 are the only subagents that received a send_message. W1 and W3–W6 received none, and every one of them was evicted as soon as a slot was needed.

    Root cause (codex-rs 0.157.1, same code on main)

    1. The eviction rule requires an empty mailbox. agent/control/residency.rs → is_unloadable() requires that the status is Completed | Errored | Interrupted, that there is no active turn, and that !input_queue.has_pending_mailbox_items(). PendingInit and Running are never unloadable.
    2. send_message queues without starting a turn. tools/handlers/multi_agents_v2/send_message.rs uses MessageDeliveryMode::QueueOnly, which sets trigger_turn = false. session/handlers.rs::inter_agent_communication enqueues the message and only starts a turn if trigger_turn is set or a durable sleep is outstanding.
    3. Queue-only messages can get stuck in a running agent's mailbox:
      • The mailbox is only read in input_queue.get_pending_input(), which runs before each model call.
      • Once the subagent emits its final-answer message (stream_events_utils.rs), defer_mailbox_delivery_to_next_turn() switches delivery to NextTurn. Turn completion (tasks/mod.rs) records only the turn's pending_input, not the session mailbox. This is what happened to W2.
      • If the turn is aborted, nothing clears the mailbox (mailbox_pending_mails is only drained by drain_mailbox_input_items()). This is what happened to W7 and W8. Neither agent made another model call between receiving the messages and being interrupted; presumably each was inside a long tool call, but that was not verified.
    4. Nothing else can drain the mailbox. Queue-only messages never wake an idle or interrupted agent (maybe_start_turn_for_pending_work requires a message with trigger_turn = true). V2 has no close_agent. The stuck message therefore pins the slot for the life of the process.
    5. The error message is misleading. AgentLimitReached is returned both by the running-turn limit (execution.rs) and by the residency limit (residency.rs), with the same text. list_agents shows nothing running, so the failure looks like a phantom concurrency leak.

    Related paths found in the source, not observed in this incident:

    • send_message to an already-evicted agent. ensure_v2_agent_loaded() reloads the agent. A reloaded session starts in PendingInit, and a queue-only message never starts a turn. The agent therefore stays PendingInit and cannot be evicted, and reloading it also evicts someone else.
    • Completion notices to a subagent parent. A child's completion notice to a subagent parent is sent with trigger_turn: false (agent/control/completion.rs). If the parent has already finished its own turn, the notice pins the parent.

    Impact

    • The number of messages sent before a spawn fails is unbounded and depends on timing. With the default of 3 slots, three unlucky messages end all further spawning in the session.
    • Staying serial does not help. The limit is on loaded agents, not on running agents.
    • A natural orchestrator pattern triggers it: "worker is slow → send a nudge → no reply → interrupt → respawn". Every pass through that loop pins another slot.
    • The in-session model misdiagnosed it. The Codex agent attributed it to the missing close_agent tool, then to concurrency semantics.

    Workarounds

    • Do not use send_message to subagents. For progress use wait_agent or list_agents. To stop a subagent, use interrupt_agent only; with an empty mailbox, an interrupted agent is evictable.
    • To unpin an agent, send it one followup_task. The new turn drains the mailbox when it starts, and once it finishes the agent is evictable again. This matches the result reported in MultiAgentV2: queue-only message to a completed agent pins a residency slot #32353. Tell the agent to ignore the stale messages, otherwise it may act on them.
    • Restart and resume the session. This should clear the pins, because residency and the mailbox are in-memory only. That conclusion is from reading the code and has not been tested.
    • Raising [agents] max_concurrent_threads_per_session only delays the failure.

    Suggested fixes

    1. Don't leave queue-only messages stranded at turn end. On turn completion or abort, record the remaining queue-only messages into history, the same way pending_input is recorded. Alternatively, persist them with the evicted agent and redeliver them on reload, so eviction does not lose them.
    2. Don't block eviction on queue-only mail. A terminal agent whose mailbox holds only queue-only messages should still be evictable, with those messages persisted.
    3. Reject send_message to terminal or evicted agents with an explicit error. Also do not reload an evicted agent for a queue-only message.
    4. Make the limit diagnosable:
      • Return distinct errors for the two limits, for example "residency limit: 3/3 loaded, 0 evictable (pending mailbox: 3, pending_init: 0)" versus "running limit".
      • Expose loaded, evictable and mailbox_pending in list_agents.
      • Log each eviction at INFO.

    Diagnosis note: the model running the Codex session (GPT-6) investigated this incident but did not identify the cause; it attributed the failure to the missing close_agent tool and to concurrency semantics. The root cause and the timeline above were located by Claude Opus 5.5, which read the codex-rs source and replayed the session's rollout files.

  9. nilswx commented on Oct 15, 2026

    @nilswx

    Current-source correction (Codex 0.162.1 source, 16 Oct 2026)

    The September 26 source audit appears stale against current main and the 0.162.1 source. In residency.rs, is_unloadable now rejects residents only when trigger-turn mailbox items remain; queue-only mail can be drained and re-enqueued in the shared runtime mailbox before the thread is unloaded. The eviction preservation path landed in #52034. The installed stock CLI reports codex-cli 0.162.1, and its matching local source contains this behavior.

    I have not run a live lifecycle reproduction against that daemon. The source change should be verified for both an already-enqueued queue-only message and the completion/admission race described above; this comment is not a claim that the runtime behavior is accepted or that the whole issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app-serverIssues involving app server protocol or interfacesbugSomething isn't workingsubagentIssues involving subagents or multi-agent features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions