Repository navigation
Completed subagent threads are not reclaimed, causing false "agent thread limit reached" with 1 Active / 12 Done #39694
Description
What version of the Codex App are you using (From “About Codex” dialog)?
26.814.41957
What subscription do you have?
Pro
What platform is your computer?
Linux 7.0.0-28-generic x86_64 x86_64
What issue are you seeing?
In a long-running Codex task, completed subagent threads stopped being reclaimed correctly after several successful waves of parallel agents.
The Subagents UI showed:
Active: 1
Done: 12
Despite this, every attempt to create another subagent failed with:
collab spawn failed: agent thread limit reached
The failure occurred both when:
- The root agent called
spawn_agent. - An active child agent attempted to spawn its own fresh child.
Representative redacted request:
spawn_agent({
agent_type: "coder",
fork_turns: "none",
task_name: "module2_cognitive_obstacle",
message: "[bounded task packet with repository paths redacted]"
})
Result:
collab spawn failed: agent thread limit reached
No subagent was created, no command was executed, and no files were changed by any rejected spawn.
Some agents that had already returned FINAL_ANSWER later appeared as pending_init:
/root running
/root/bootstrap_guidance_worktrees pending_init
/root/module2_cognitive_pose pending_init
/root/module3_gt_firewall running
...other agents completed
Attempting to interrupt these stale entries returned:
{"previous_status":"pending_init"}but did not close or remove them.
I then sent each stale agent a no-op cleanup follow-up. Both returned:
cleanup complete
Their status changed from pending_init to completed, but new spawn_agent calls still failed with the same thread-limit error.
When the only active child later completed, exactly one new spawn succeeded. An immediate second spawn again failed with:
collab spawn failed: agent thread limit reached
This indicates that the scheduler’s open-thread count did not match the UI Active count.
The official [Subagents documentation](https://learn.chatgpt.com/docs/agent-configuration/subagents) says Codex handles closing agent threads and describes agents.max_concurrent_threads_per_session as a cap on concurrently open spawned-agent threads, excluding the primary thread. The observed behavior appears inconsistent with those semantics.
What steps can reproduce the bug?
- Enable multi-agent support with a concurrency limit such as:
[agents]
enabled = true
max_concurrent_threads_per_session = 5-
Start a long-running goal/task.
-
Spawn four independent subagents concurrently.
-
Wait until all four return final results and appear under Done.
-
Spawn additional waves of fresh subagents. In this session, several later waves initially worked correctly.
-
Continue until the UI contains approximately 10–12 Done subagents.
-
Leave one child agent running.
-
Attempt to spawn another fresh agent:
spawn_agent({
agent_type: "coder",
fork_turns: "none",
task_name: "another_independent_task",
message: "[redacted bounded task]"
})
- Observe:
collab spawn failed: agent thread limit reached
-
Inspect the agent list. Some agents that already returned final results may appear as
pending_init. -
Attempt to close them using
interrupt_agent. -
If they remain registered, send a no-op follow-up so they return another final result.
-
Retry
spawn_agent. In the affected session, the spawn still failed even after the stale agents becamecompleted. -
Wait for the one active child to finish. One new spawn may then succeed, while an immediate second spawn fails again.
Session/task ID:
01a01a09-e10b-79a3-b71e-15f531937900
Token-limit and context-window usage:
Not captured. No token-limit, context-window, or compaction warning was shown when the scheduling failure occurred.
Additional reproduction detail:
- Earlier waves of four parallel agents and subsequent replacement waves succeeded.
- The failure appeared only after the long-running task accumulated many completed agent threads.
- Raising the persisted concurrency setting from 5 to 10 during the running task did not repair the affected session. This configuration reload behavior may be expected and is not the primary bug.
What is the expected behavior?
When a subagent returns its final result:
- Its agent thread should transition to a completed and reclaimable/closed state.
- It should not later reappear as
pending_init. - Done agents should not consume active concurrency indefinitely.
- The UI Active count and the scheduler’s open-thread count should agree.
- A new subagent should be spawnable whenever the number of concurrently open child threads is below
agents.max_concurrent_threads_per_session. - Asking Codex to close a completed thread should actually release its scheduler capacity.
- If spawning is rejected, the error should report the relevant counts, for example:
configured_limit
open_spawned_threads
active_threads
completed_but_open_threads
pending_init_threads
Additional information
Environment details:
OS: Ubuntu 24.04 x86_64
ChatGPT/Codex Desktop package: 26.814.41957
Embedded Codex CLI: 0.148.0-alpha.15
Standalone Codex CLI: 0.147.0
Task type: long-running /goal workflow
Observed lifecycle anomaly:
FINAL_ANSWER
↓
completed
↓
unexpected pending_init
Cleanup attempts:
interrupt_agent(pending_agent)
→ previous_status: pending_init
→ agent remained registered
no-op followup_task(pending_agent)
→ FINAL_ANSWER: cleanup complete
→ state became completed
→ scheduler capacity was still not reclaimed
Impact:
- Parallel execution degraded to effectively one child at a time.
- The UI suggested available capacity that the scheduler would not allocate.
- Workflows requiring fresh agents could no longer follow their configured orchestration rules.
- Restarting or creating a new task was the only reliable way to rebuild the agent registry.
- Rejected spawn attempts caused no repository or filesystem changes.
Possible cause, based only on observed behavior:
- completed agent instances may remain registered as scheduler-open threads;
FINAL_ANSWER,completed,pending_init, UI Active/Done, and scheduler-open states may not be synchronized;- reclamation may degrade after repeated waves in a long-running task.
Suggested attachments:
- Screenshot showing
1 Active / 12 Done. - Redacted
list_agentsoutput. - The exact
agent thread limit reachederrors. - Logs showing agents returning
FINAL_ANSWERbefore appearing aspending_init. - A sanitized
config.toml. - Session/task ID shown above.
Activity
github-actions commentedon Aug 20, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
dajiaohuang commentedon Aug 20, 2026
A current-main source check at
39073ca3suggests the immediate admission failure is residency pressure, not necessarily leaked active-execution guards. V2 has separate active-turn and resident-slot gates, but both return the sameAgentLimitReached. The residency LRU counts loaded residents and only evictsCompleted/Errored/Interruptedagents whose turn is cleared and mailbox is empty;PendingInitis not evictable. Reloaded sessions start atPendingInit, resume reconstruction restores onlyInterrupted, andinterrupt_agentsubmits an interrupt and returns the previous status—it does not close or evict the resident.That makes the reported one-for-one behavior source-consistent: when the sole running child becomes terminal, one evictable resident appears, one spawn succeeds, and the resident set can be blocked again. The exact event that reloaded the affected children still needs logs; queue-only delivery is one public path that reloads without immediately starting a turn.
This appears closest to #34518 (including an independent
0.148.0-alpha.15pending_initreproduction) and the broader #22779. #38805/#38247 are related persistence/runtime-retention reports, but not as exact for admission accounting. A regression should fill the resident cap, evict and reload a terminal child without starting a turn, exerciseinterrupt_agentandfollowup_task, and assert both spawn admission and the UI/list status projection. The error should also distinguish active-cap from residency-cap rejection and report blockers.pomazanbohdan commentedon Aug 23, 2026
The source-level residency diagnosis matches a Windows reproduction where child rollouts were terminal but parent edges stayed open; after compaction a completed child returned as pending_init, and the next spawn hit the same limit. Cleanup must distinguish live execution from resident/edge state; this bridges #34518 and #22779 with the UI projection bug in #37042.
theosp commentedon Aug 24, 2026
This report was written by the Codex agent that designed the tests and independently inspected the coordinator and child-task records.
Summary
After updating Codex Desktop, two controlled tests of completed-subagent slot reclamation succeeded:
No subagent creation was refused, and no
agent thread limit reachederror occurred.Important limitation
Unfortunately, we did not run these exact controlled tests before updating Codex. Therefore, we cannot confirm with 100% certainty that the update fixed the problem or establish the update as its cause.
What we can establish is:
Versions
Previously observed environment:
26.818.4150969620.149.0-alpha.4.1Tested environment after updating:
26.818.6180970190.149.0-alpha.4.3The tests ran inside Codex Desktop, so the bundled CLI/runtime is the relevant one.
Test 1: Sequential reclamation
Prompt
Result
SLOT-1-OK.rather thanSLOT-1-OK; the additional period is unrelated to slot reclamation.This demonstrates that normal completion no longer exhausts fresh-child capacity over six sequential creations.
Test 2: Reclamation between waves
Prompt
Result
Because these children performed extremely short tasks, some Wave 2 children completed while the remaining dispatches were being issued. The test nevertheless confirmed that fresh child creation continued after a completed three-child wave without encountering the previous limit error.
Conclusion
The updated environment successfully passed both:
This is strong evidence that the observed completed-agent slot-reclamation problem is not currently reproducing.
It is not proof that every long-running, interrupted, approval-blocked, or unusually complex subagent lifecycle is fixed. Most importantly, because these controlled tests were not run before the update, the result cannot establish with certainty that the update caused the improvement.
theosp commentedon Aug 25, 2026
Follow-up reproduction on the updated build
Environment:
26.818.618090.149.0-alpha.4.301a03411-e72a-73a2-aae2-97872a621dabA long-running multi-agent session reproduced the false thread-limit symptom with a more specific lifecycle sequence:
send_messageto child A while A was still running.collab spawn failed: agent thread limit reached.completed.This strongly suggests a completed resident pinned by the near-completion mailbox race described in #32353, rather than cumulative child exhaustion. Sixteen fresh children had been created over earlier waves, but there is no evidence that 16 is a lifetime limit.
This does not contradict my previous controlled tests: those deliberately used no messages, follow-ups, or interruptions, so they did not exercise this lifecycle.
The-skomoroh commentedon Sep 13, 2026
Additional 0.153.1 reproduction: completion arrives before residency is reclaimable
After one of three child agents reported completion, only two child turns were running. An immediate fresh
spawn_agentfailed with:A
followup_taskto the just-completed resident succeeded. This distinguishes fresh-residency admission from active-execution capacity.At current main
1715e550, the ordering explains this window:RunningTaskowns the execution guard (state/turn.rs).on_task_finishedtakes and dropsRunningTask, releasing that guard (tasks/mod.rs).TurnComplete(tasks/mod.rs);send_eventforwards the terminal result to the parent before returning (session/mod.rs).active_turncleared (tasks/mod.rs).active_turn == None, and an empty mailbox (residency.rs).The parent can therefore react to a valid completion while the child is terminal but not yet evictable. At full residency, the replacement receives the same generic
AgentLimitReachedused for execution-cap rejection.Expected contract: absent an intervening restart or follow-up, once a terminal result for an empty-mailbox child is delivered to its parent, that child should already be reclaimable by a concurrent fresh spawn. If terminal delivery must remain earlier, the runtime needs a machine-readable, retryable residency-ready transition after full quiescence rather than treating completion itself as capacity-ready. Admission failures should distinguish execution-capacity exhaustion from residency exhaustion. Any configured/in-use/resident/evictable counts are point-in-time diagnostics, not an availability guarantee.
Alex-R-A commentedon Sep 17, 2026
Additional reproduction on a current release:
Environment:
Observed sequence:
This supports a mismatch between the visible agent tree and the scheduler or residency registry. It does not establish whether the hidden blocker was resident state, mailbox state, or stale accounting.
dajiaohuang commentedon Oct 1, 2026
Additional reproduction on Codex Desktop for Windows (2026-10-01; app version not captured):
pending_init.interrupt_agentreturnedprevious_status: pending_init, and the child remained in that state. A subsequent spawn attempt failed withagent thread limit reached.FINAL_ANSWER. The child then appeared ascompleted, and a later fresh agent spawn succeeded while two other child agents were still running.In this session, the follow-up did release capacity, unlike the reproduction described in this issue where completed follow-ups still did not. No app restart was needed. The app version was not recorded.
fusengine commentedon Oct 4, 2026
Same bug, worse numbers. codex-cli 0.160.0, Codex desktop,
multi_agent_v2enabled (max_concurrent_threads_per_session = 6).spawn_agentfails withcollab tool failed: agent thread limit reached— 391 occurrences in a single session from 2026-10-01 to 2026-10-04.Blunt feedback from a paying user: if you're going to charge subscriptions of up to $500/month, ship a client that works and an agent smart enough not to deadlock itself. Completed threads must be reclaimed automatically, or at least closable.
pomazanbohdan commentedon Oct 4, 2026
I haven't had these problems for over a month. Deleted all old sessions, started new ones I asked the coder if the bootstrap or update was going well, and he found from the logs that the conversion was not happening because the processes were hanging.Archive all old sessions and delete them, if there is an error then let the code clean it up.
dajiaohuang commentedon Oct 7, 2026
Thanks for the additional reproductions and the blunt feedback, @fusengine. The 440 completed residents with 0 active and 391 spawn failures is an extreme but consistent manifestation of the residency reclamation issue: completed agents are not being evicted from the resident set even after their turns finish and mailboxes are empty, so fresh spawns hit the generic 'agent thread limit reached' error regardless of actual execution capacity.
@pomazanbohdan: archiving and deleting old sessions is a valid user-side workaround for sessions that have already accumulated stale residents, though it does not address the root cause in the residency/eviction logic. In my own Windows reproduction earlier this week, sending a follow-up to the stuck 'pending_init' child did release capacity (after which a fresh spawn succeeded), but that workaround is not reliable as fusengine and Alex-R-A have reported cases where even follow-ups or interrupts do not clear the block.
The core issue remains as The-skomoroh identified in the source ordering: terminal results are delivered to the parent before the child is marked evictable (active_turn cleared and mailbox empty), so a concurrent fresh spawn can fail spuriously, and there is no mechanism to retry eviction or distinguish residency exhaustion from active execution capacity limits.
fusengine commentedon Oct 7, 2026
multi_agent_v2: no way to release completed subagents —
close_agentships in the binary but is only exposed in v1Version: codex-cli 0.160.1 (Homebrew cask) · macOS arm64 · model
gpt-6-astra(multi_agent_version: v2)Config
What happens
Completed.spawn_agentfails withagent thread limit reached.spawn_agent,send_message,wait_agent,list_agents,interrupt_agentare exposed.interrupt_agentdoes not free the slot.Why this looks like a design gap, not just a bug
Strings in the 0.160.1 binary show:
core/src/tools/handlers/multi_agents/close_agent.rsexists.core/src/tools/spec_plan.rs),close_agentis grouped with the v1-only set:multi_agent_v1→send_input,resume_agent,close_agent.close_agentdescription itself says: "Completed agents remain open and count toward the concurrency limit until closed."v2 resident eviction task failed,failed to shut down v2 resident thread before unloading).So v2 removed the manual release path and depends entirely on eviction. When eviction does not happen (see #39694, #50584, #32353), the session is stuck with no recovery tool, while the code to recover already exists.
The docs make it worse: the subagents page says "Ask Codex directly to … close completed subagent threads", which is impossible under v2.
Expected
Any one of these would fix it:
close_agent(or an equivalentrelease_agent) in v2 as a fallback when a completed agent is not evicted.max_concurrent_threads_per_session, and evict synchronously before rejecting a spawn.pending_init…).Related
#39694 (same config: 0 active / 440 done, still blocked) · #50584 (false limit below the configured value) · #32353 (queue-only message pins a completed agent) · #33447 (v2 silently ignores
agents.max_threads)fusengine commentedon Oct 7, 2026
Root cause found (reproducible) + workaround —
thread_spawn_edgesrows are never closed in v2Follow-up to my comment above. Same setup (codex-cli 0.160.1,
multi_agent_v2,max_concurrent_threads_per_session = 6).Observation in
~/.codex/state_5.sqlite:Every subagent ever spawned is still
open— including ones from previous days. The stuck session alone had 118openedges, allCompletedfrom the agent's point of view.Per
codex-rs/agent-graph-store/src/types.rs,Openmeans "the child thread is still live or resumable as an open spawned agent". Under v2 nothing ever writesClosed(noclose_agenttool;interrupt_agentdoes not touch the edge), so completed agents keep counting.Experiment (both directions):
UPDATE thread_spawn_edges SET status='closed' WHERE parent_thread_id='<session>' AND status='open';(118 rows) → resume the session → spawns work again.openedges = 5 subagent slots (6 incl. root) → the 6th spawn fails withagent thread limit reached, although all 5 are completed and one was interrupted.So the limit is driven by
openspawn edges, and v2 has no path that closes them.Workaround (Codex must be fully quit — app and CLI; back up first):
Suggested fix: set the edge to
Closedwhen a v2 child reaches a terminal state (Completed/Errored/Interrupted) or is evicted, and/or exposeclose_agentin v2.fusengine commentedon Oct 7, 2026
Correction + working workaround: open
thread_spawn_edgesare reloaded on resumeCorrection to my previous comment. I wrote that "the limit is driven by open spawn edges". That is wrong for a running session. Watching
~/.codex/state_5.sqlitelive, the same session later reached 9openedges with 5 subagent slots and kept spawning without error, so v2 does free slots in memory, but it never writes the edge back toclosed.What the data does support:
Timeline from the session rollout (codex-cli 0.160.1, Codex Desktop,
multi_agent_v2,max_concurrent_threads_per_session = 6):openedges; resuming →agent thread limit reached.closed→ resume → spawns work (5 agents, then a 6th, 7th… up to 9openedges, no error).openedges left → resumed session →collab tool failed: agent thread limit reachedat 11:40 and 11:43.So the leak is on the persistence/resume path. The workaround below has unblocked the session every time.
Workaround (Codex fully quit: app and CLI)
Run it before relaunching Codex after any long multi-agent session, not only once you are blocked. It only changes the edge status: thread history is kept. I wrapped it in a small shell function that refuses to run while Codex is open, backs up the DB and prints the restore command.
Suggested fix
Closedwhen a v2 child reaches a terminal state (Completed/Errored/Interrupted) or is evicted.close_agentin v2 as a manual fallback (it ships in the binary but is only in the v1 tool set).nilswx commentedon Oct 15, 2026
Current-source qualification (rust-v0.162.1, tag commit
7a078f6bb4ce0dc2a81408b334ce52979d92c603)The open-edge count alone does not appear to consume the V2 resident cap on this tag.
restore_v2_agent_metadatarestores child identity/path metadata withregistry.reserve_spawn_slot(None)and explicitly does not reopen runtimes (spawn.rs). V2 spawn admission uses the separateV2Residencycounter, which restoration does not repopulate (residency.rs,spawn.rs). The reported 0.148 alpha/0.147 behavior therefore still deserves runtime reproduction, but the specific theory that passive open edges alone fill V2 residency is stale for 0.162.1. Restored paths can still reserve names, and loading a child later consumes a normal resident slot.I did not run this lifecycle against a live daemon; this is source-only and does not rule out the reported admission/status mismatch or the terminal-result versus
active_turn-clear race.