Repository navigation
MultiAgentV2: queue-only message to a completed agent pins a residency slot #32353
Description
What version of Codex CLI is running?
Observed with codex-cli 0.144.0 bundled in Conductor. I also checked codex-cli 0.144.1 and current main (5c19155cbd93bfa099016e7487259f61669823ff); the relevant v2 residency and message-delivery logic is unchanged.
What platform is your computer?
Darwin 25.5.0 arm64 arm
Surface: Codex app-server / MultiAgentV2, accessed through Conductor 0.74.0.
What issue are you seeing?
A completed MultiAgentV2 subagent can indefinitely occupy a resident thread slot when it has a pending queue-only mailbox message. Once the other resident slots are active, a new spawn_agent fails even though the number of actively running subagents is below the configured limit:
collab spawn failed: agent thread limit reached
This is not cumulative thread exhaustion. In the observed session, spawning succeeded again later after the pinned mailbox was consumed. The same AgentLimitReached error currently represents active concurrency exhaustion and residency exhaustion, so the parent cannot tell which condition occurred or which agent is blocking eviction.
What steps can reproduce the bug?
Use a small v2 capacity so the behavior is visible:
[features.multi_agent_v2]
max_concurrent_threads_per_session = 4The value includes the root agent, leaving three subagent resident slots.
- Spawn subagent A and let it reach
Completed. - Call
send_messagetargeting A. In v2 this is queue-only and does not trigger a new turn. - Spawn subagents B and C with tasks that remain active.
- Attempt to spawn subagent D.
- D fails with
collab spawn failed: agent thread limit reached, although only B and C are active. - Call
followup_taskon A so it wakes, consumes its mailbox, and completes again. Retry the spawn; it can now succeed.
The production occurrence was the race-shaped variant: the root called send_message near A's completion boundary. A completed about one second later, leaving the queue-only message pending. With B and C active, spawning D failed.
What is the expected behavior?
A queue-only message should not leave a completed agent as a hidden, indefinitely non-evictable resident.
Any of these behaviors would avoid the deadlock:
- reject
send_messageto an already completed agent and tell the caller to usefollowup_task; - automatically trigger a follow-up turn when a queue-only message crosses the target's completion boundary;
- persist the mailbox independently so the completed thread can be unloaded and resumed later;
- expose an explicit close/drain operation for completed v2 agents.
At minimum, AgentLimitReached should distinguish active concurrency from residency pressure and report data such as max, residents, active, and pending_mailbox_blockers with an actionable recovery hint.
Additional information
Current main makes the interaction explicit:
send_messageisQueueOnly, whilefollowup_taskisTriggerTurn.- A completed v2 resident is unloadable only if its mailbox has no pending items.
- If no resident can be unloaded, admission returns the generic
AgentLimitReached.
Related but distinct:
- Completed subagents continue to count against thread limit #22779 covers completed subagents continuing to count against the older/open-thread quota. This report is specifically about v2 residency, where ordinary completed agents are evictable but completed agents with pending queue-only mail are not.
- Main agent needs subagent capacity/status preflight before spawn_agent attempts #23479 requests capacity/status preflight; reporting mailbox blockers would address this failure mode.
- Plan Mode should allow safe cleanup of completed subagents #24508 requests safe cleanup of completed subagents, but v2 currently has no direct close tool in the exposed delegation toolset.
Increasing features.multi_agent_v2.max_concurrent_threads_per_session reduces the frequency but does not fix the pinned-mailbox lifecycle bug.
Activity
Yurdos-r commentedon Jul 21, 2026
Independent reproduction from Codex Desktop (2026-07-21)
I reproduced the admission failure with a controlled baseline and a pending-mailbox variant.
Environment
Baseline: ordinary completed agents were reclaimable
A_OK,B_OK, andC_OKand reachcompleted.Result: D's
spawn_agentreturned successfully without a noticeable wait, and D completed withD_OK.This indicates that ordinary completed agents alone did not block a later spawn in this runtime.
Variant: a queue-only message pinned the completed resident
A_OK.STATUS_PROBEto completed A using queue-onlysend_message. It did not trigger a new turn.The G spawn failed immediately with the exact output:
At that moment only E and F were actively running. A was completed, but it had the pending queue-only mailbox item. The controlled difference from the successful baseline was the pending mailbox on A.
I then called
followup_taskon A; A consumed the queued work and returnedA_DRAINED. E and F also completed. A subsequent retry could not be emitted because the isolated test task encountered a separate tool-routing failure, so I am not claiming that the recovery path was independently verified here.What this reproduction supports
completedresidents were evictable/reclaimable in the baseline.completedresident with pending queue-only mail continued to consume residency capacity.AgentLimitReachederror as true active-concurrency exhaustion.This independently supports the pending-mailbox residency mechanism described in the issue and the need for an actionable error that distinguishes active concurrency from residency pressure.
KNaiFen commentedon Aug 5, 2026
I audited this against current
mainatc38a60ded2ffbad510f8b512e2902f0128e998e8. The reported residency mechanism is still present, and there is an important race constraint for a fix.Why a tool-handler status precheck is insufficient
handle_message_string_toolresolves and loads the receiver, then submits anOp::InterAgentCommunication. The actual mailbox insertion happens later insession::handlers::inter_agent_communication:A
Completedcheck inmessage_tool.rsalone can pass while the target is still running, followed immediately by target completion before the queued op reachesenqueue_mailbox_communication. That recreates the same completed-plus-pending-mailbox blocker.Current task completion calls
maybe_start_turn_for_pending_work, but that helper intentionally starts only fortrigger_turnmail (or durable sleep). Queue-only mail therefore remains pending, andresidency::is_unloadablerejects the resident becausehas_pending_mailbox_items()is true.Invariant a fix should enforce
After the target reaches a terminal state, queue-only mail must not become an indefinite residency blocker. The atomic decision needs to live at the actual mailbox admission/completion boundary, not only in the model-facing tool handler.
Two viable semantics appear to require maintainer direction:
use followup_taskerror. This needs an acknowledgement path from the session handler back to the tool call; the current op submission only confirms that the op was accepted by the session loop.Regression coverage
A focused test should cover both orderings:
send_message(A).AgentLimitReachedcaused by a hidden queue-only blocker.Per the invitation-only contribution policy, I will not open an unsolicited PR. If a maintainer selects the intended delivery semantic and invites a contribution, I can implement the narrow fix and integration test.
wuyak commentedon Aug 9, 2026
This issue also exposes a message-lifecycle boundary that is currently difficult for the caller to observe:
I opened #37744 for the complementary protocol gap: a parent should be able to distinguish
queued,consumed,applied,rejected,superseded, andunknown, rather than treating a completedsend_messagecall as proof of application.At a terminal boundary, the runtime should define one deterministic behavior: reject queue-only delivery and direct the caller to a follow-up turn, promote the message into a new turn, or externalize the pending mailbox so it does not pin residency. Whichever behavior is selected, the resulting delivery and reclaim state should be observable.
theosp commentedon Aug 25, 2026
Corroborating production reproduction
Observed on Codex Desktop
26.818.61809with bundled CLI0.149.0-alpha.4.3in session01a03411-e72a-73a2-aae2-97872a621dab.The parent called queue-only
send_messageon child A while A was running; A completed shortly afterward. Two fresh siblings then started, but a third fresh spawn failed withagent thread limit reached. At failure, the list showed the root plus the two siblings running and A stillcompleted. Callinginterrupt_agenton A reported its prior completed status but did not free admission, and the third spawn failed again.This closely matches the race-shaped reproduction described here: a message crossing the completion boundary appears to leave the completed child resident and non-evictable. It also explains the broader symptom reported in #39694. The session had previously created 16 fresh children over multiple waves, but the evidence supports residency pressure—not a cumulative 16-child limit.
lucasgnz commentedon Sep 9, 2026
Additional reproduction and local prototype validation on 0.153.4 (macOS arm64)
A Codex-assisted investigation reproduced this specific pending-mailbox case against the official rust-v0.153.4 sources (3d2ee51), using the existing mocked Responses integration harness rather than production conversations.
Minimal integration scenario:
Step 4 fails on the unmodified tag: A cannot be evicted because is_unloadable requires an empty mailbox. The new integration test fails on the original source, including the automatic retry.
A local prototype makes the scenario pass by retaining an
Arc<InputQueue>in the session-scoped agent registry alongside evicted environment state. A reloaded session reuses that queue; cached eviction state is cleared only after successful reload. Terminal, idle residents with queue-only mail become evictable, while trigger-turn mail and durable-sleep sessions remain protected. This retains the queue within the same application process; it does not add crash-durable mailbox storage.The same eviction path also needs reservation/cancellation coverage: a candidate temporarily removed from the LRU during awaited shutdown must not appear as free capacity to another reservation, and cancellation must restore or finish reclaiming that candidate. The prototype serializes residency reservations and restores candidates on cancelled eviction, including retry after shutdown.
Validation with Rust 1.97.1:
As checked on September 8, the residency and registry files are unchanged in rust-v0.154.0-alpha.8 and main at 5ac0b87. The relevant queue-only admission condition also remains present.
Sharing this as reproduction and design evidence under the contribution policy. The prototype has not been deployed into the Desktop runtime or validated across platforms. No production logs, private paths, task identifiers, or project content are included.
2217173240 commentedon Sep 23, 2026
Additional production trace on bundled CLI 0.155.0-alpha.16 (macOS arm64)
This is corroborating trace evidence for the completion-boundary variant, not a claim of a new deterministic reproduction or a confirmed in-memory mailbox inspection. Agent names below are anonymized.
Environment:
Observed sequence on 2026-09-23 (UTC):
send_messageto worker A.task_complete.send_messagereturns successfully.spawn_agentfails withcollab spawn failed: agent thread limit reached.completed, while ordinary completed siblings disappear and are replaced.completed.completedin the resident list, about 86 minutes after its task completed.No subsequent
followup_taskorinterrupt_agenttargeting A appears in the inspected interval. The child rollout contains no additional turn after the recorded completion. This is consistent with a stranded queue-only message, although the private in-memory mailbox and residency permit state were not inspected, so the causal mechanism remains inferred from the trace and source.There was a separate local configuration rollback during this broader investigation. I excluded that from the lifecycle claim above: the affected runtime already announced seven slots throughout this interval. This report does not claim a model-specific concurrency limit or that changing the limit fixes the lifecycle issue.
Source check against main at
40eac3ce8a0c10cbcb9db910d529355eb2f8fc09:is_unloadablestill requires!has_pending_mailbox_items()for a terminal idle worker.rust-v0.155.0-alpha.16source has the same pending-mailbox unloadability restriction.Related merged PR #47240 protects queued submissions and makes eviction/removal race-safe. It should not be assumed to resolve this terminal queue-only mailbox case; the restriction above remains on main. I have not run a reproduction against a main build.
A useful regression case remains the already-described controlled completion/admission interleaving, checking that messages are preserved and that settled terminal workers do not indefinitely block replacement admission. An observable distinction between active capacity and non-evictable mailbox residency would also help operators diagnose this without interrupting healthy siblings.
No customer content, credentials, local paths, raw rollouts, or private task identifiers are included. This report was prepared with Codex assistance from local trace inspection; the live failing task was not restarted or forcibly terminated for this report.
chunxiaoyi commentedon Sep 26, 2026
MultiAgentV2: queue-only
send_messagestranded in a subagent's mailbox pins its residency slot for the rest of the session →agent thread limit reachedin a strictly serial workflowRelated: #32353 (same root cause, idle-target variant), #39694 (same symptom)
Summary
In a strictly serial orchestration (never more than one subagent running at a time),
spawn_agentstarted failing withcollab spawn failed: agent thread limit reachedafter 8 successful spawns. At the time of failure, no subagent was running.The 3 resident slots were held by the only 3 subagents in the session that had ever received a
send_messagefrom the root. In each case at least one of those queue-only messages was never drained by the recipient: the recipient either finished its turn a few seconds after the message arrived, or was interrupted while the message was still queued. A subagent whose mailbox is not empty never passesis_unloadable, so it can never be evicted. After the third such subagent, every later spawn failed. The other 5 subagents never received a message, and all of them were evicted normally.This extends #32353. The target does not have to be idle when the message is sent. A message sent to a running subagent can also get stuck, for either of two reasons:
Environment
gpt-6-sol(catalog reportsmulti_agent_version = v2,tool_mode = code_mode_only)max_concurrent_threads_per_session = 4→ 3 subagent slots; no overrides inconfig.tomlwait_agent, collects output, then spawns the next. Never more than one subagent running.Timeline
2026-09-26, times in UTC+08:00. Subagents are renamed
W1–W9in spawn order.Evidence used:
inter_agent_communication_metadatarecord withtrigger_turn: false; an unconsumed one leaves nothing.agent/control/residency.rs. The replay matches all 8 successful and 3 failed spawns.send_message→ W2 (W2 running). Consumed by W2 at 18:17:39send_message→ rootsend_message→ W2 (W2 still running)send_message→ W7 (running). Never consumedsend_message→ W7 (running). Never consumedinterrupt_agent→ W7. W7 turn aborted (interrupted) with 2 messages still queued. W7 is now pinnedsend_message→ W8 (running). Never consumedsend_message→ W8 (running). Never consumedinterrupt_agent→ W8. W8 turn aborted (interrupted) with 2 messages still queued. W8 is now pinnedcollab spawn failed: agent thread limit reachedlist_agents: no subagent runningagent thread limit reachedagent thread limit reached. Logged atERRORbycodex_core::tools::routerState of the resident set at 18:44:04:
is_unloadableRunning subagents: 0. The running-turn limit was not reached; the residency limit was.
Across the whole session, W2, W7 and W8 are the only subagents that received a
send_message. W1 and W3–W6 received none, and every one of them was evicted as soon as a slot was needed.Root cause (codex-rs 0.157.1, same code on
main)agent/control/residency.rs→is_unloadable()requires that the status isCompleted | Errored | Interrupted, that there is no active turn, and that!input_queue.has_pending_mailbox_items().PendingInitandRunningare never unloadable.send_messagequeues without starting a turn.tools/handlers/multi_agents_v2/send_message.rsusesMessageDeliveryMode::QueueOnly, which setstrigger_turn = false.session/handlers.rs::inter_agent_communicationenqueues the message and only starts a turn iftrigger_turnis set or a durable sleep is outstanding.input_queue.get_pending_input(), which runs before each model call.stream_events_utils.rs),defer_mailbox_delivery_to_next_turn()switches delivery toNextTurn. Turn completion (tasks/mod.rs) records only the turn'spending_input, not the session mailbox. This is what happened to W2.mailbox_pending_mailsis only drained bydrain_mailbox_input_items()). This is what happened to W7 and W8. Neither agent made another model call between receiving the messages and being interrupted; presumably each was inside a long tool call, but that was not verified.maybe_start_turn_for_pending_workrequires a message withtrigger_turn = true). V2 has noclose_agent. The stuck message therefore pins the slot for the life of the process.AgentLimitReachedis returned both by the running-turn limit (execution.rs) and by the residency limit (residency.rs), with the same text.list_agentsshows nothing running, so the failure looks like a phantom concurrency leak.Related paths found in the source, not observed in this incident:
send_messageto an already-evicted agent.ensure_v2_agent_loaded()reloads the agent. A reloaded session starts inPendingInit, and a queue-only message never starts a turn. The agent therefore staysPendingInitand cannot be evicted, and reloading it also evicts someone else.trigger_turn: false(agent/control/completion.rs). If the parent has already finished its own turn, the notice pins the parent.Impact
close_agenttool, then to concurrency semantics.Workarounds
send_messageto subagents. For progress usewait_agentorlist_agents. To stop a subagent, useinterrupt_agentonly; with an empty mailbox, an interrupted agent is evictable.followup_task. The new turn drains the mailbox when it starts, and once it finishes the agent is evictable again. This matches the result reported in MultiAgentV2: queue-only message to a completed agent pins a residency slot #32353. Tell the agent to ignore the stale messages, otherwise it may act on them.[agents] max_concurrent_threads_per_sessiononly delays the failure.Suggested fixes
pending_inputis recorded. Alternatively, persist them with the evicted agent and redeliver them on reload, so eviction does not lose them.send_messageto terminal or evicted agents with an explicit error. Also do not reload an evicted agent for a queue-only message.loaded,evictableandmailbox_pendinginlist_agents.INFO.Diagnosis note: the model running the Codex session (GPT-6) investigated this incident but did not identify the cause; it attributed the failure to the missing
close_agenttool and to concurrency semantics. The root cause and the timeline above were located by Claude Opus 5.5, which read thecodex-rssource and replayed the session's rollout files.nilswx commentedon Oct 15, 2026
Current-source correction (Codex 0.162.1 source, 16 Oct 2026)
The September 26 source audit appears stale against current
mainand the 0.162.1 source. Inresidency.rs,is_unloadablenow rejects residents only when trigger-turn mailbox items remain; queue-only mail can be drained and re-enqueued in the shared runtime mailbox before the thread is unloaded. The eviction preservation path landed in #52034. The installed stock CLI reportscodex-cli 0.162.1, and its matching local source contains this behavior.I have not run a live lifecycle reproduction against that daemon. The source change should be verified for both an already-enqueued queue-only message and the completion/admission race described above; this comment is not a claim that the runtime behavior is accepted or that the whole issue is resolved.