Repository navigation
Claude wake latch: a wake queued behind a held run swallows later wakes, so idle-launched subagents stay Running #22033
Description
A Claude native subagent launched while the root is idle can be discovered from task_progress during a later user turn, yet stay Running after its final task_notification arrives.
Observed in the macOS desktop nightly 0.1.1-nightly.20261018.3331. The deterministic replay runs on Windows, which has the same nightly installed, and fails against both that nightly's adapter source and main at cb0553ea39.
Minimal SDK replay, using the production Claude adapter:
- Start and finish an initial root user turn.
- Deliver
system/task_startedfor one backgroundlocal_agentwhile the root is idle. Leave its buffered continuation undrained. - Start an ordinary user turn. Deliver
system/task_progressfor that task. The adapter creates the subagent from this progress event, with a fallback title and a late start time. - Finish that user turn.
- Deliver the task's final
system/task_notification {status: completed, summary: "Audit finished."}while the root is idle. - Read the latest
subagent.updated: it remainsrunning, with no completion timestamp/result.
Expected: the authoritative final notification settles the registered subagent, even when its older launch frame remains in the wake buffer.
No SendMessage resume, nested agent, provider switch, process replacement, or background shell is needed. The external SDK stream is stubbed in this deterministic replay; this is not a live authenticated Claude CLI reproduction. The matching production incident has both raw task_updated {status: completed} and task_notification completed in the provider log while the persisted row stays running.
The failing gate is in ClaudeAdapterV2.ts at cb0553ea39: hasBufferedStart prevents the idle outcome from being applied, even though progress already registered that same launch during the later user turn. The normal user-turn path does not drain the wake buffer. The guard came from 265fb7000f (#18778), which already exists in the affected nightly. A raw completion signal reaches T3; the SDK is not dropping it in this case.
Related: #17809, #18910 and #20430. The stuck continuation gate in #20430 can leave such a start undrained, but this replay needs no held queued run. #16610 concerns interim completed notifications followed by self-resume; that is a separate observation. #21724/#21725 reconcile unregistered tasks after process replacement and do not remove this guard for registered tasks.
Reproduction added to the existing ClaudeAdapterV2.test.ts suite:
vp test run apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts -t 'settles an idle-launched subagent discovered'
# expected 'running' to equal 'completed'
# 1 failed, 299 skippedThe same assertion failure occurs with only the adapter source replaced by the exact installed nightly tag, and with current-main adapter source. All task names, IDs, prompts and output in the test are synthetic. No user transcripts, database, screenshots or private project data are attached.
Implemented/investigated with GPT-6.1 Sol through Codex in T3 Code.
Repro correction: the first short test waited for SDK processing but did not wait for the separate event collector. The test-only patch below adds an ordered provider-thread roster receipt after the final notification. That receipt arrives on both broken and fixed adapters, so the status assertion is independent of whether a completion event was emitted. This strengthened test still fails on cb0553ea39 with actual running, expected completed; the candidate fix passes it, the SQLite persistence case, and the existing genuine-resume case. Broader focused checks and review are still running.
Test-only reproduction patch against cb0553e
Apply this patch, then run the command above. It changes only the existing test file.
diff --git a/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts b/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
index 3b09094aa1..98c6018f5d 100644
--- a/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
+++ b/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
@@ -2538,6 +2538,10 @@ describe("ClaudeAdapterV2 background wake turns", () => {
yield* Queue.unbounded<
Extract<ProviderAdapter.ProviderAdapterV2Event, { type: "subagent.updated" }>
>();
+ const providerThreadReceipts =
+ yield* Queue.unbounded<
+ Extract<ProviderAdapter.ProviderAdapterV2Event, { type: "provider_thread.updated" }>
+ >();
const systemNoticeReceipts =
yield* Queue.unbounded<
Extract<ProviderAdapter.ProviderAdapterV2Event, { type: "turn_item.updated" }>
@@ -2646,6 +2650,9 @@ describe("ClaudeAdapterV2 background wake turns", () => {
if (event.type === "subagent.updated") {
yield* Queue.offer(subagentReceipts, event);
}
+ if (event.type === "provider_thread.updated") {
+ yield* Queue.offer(providerThreadReceipts, event);
+ }
if (event.type === "turn.terminal") {
yield* Queue.offer(terminalReceipts, event);
}
@@ -2687,6 +2694,7 @@ describe("ClaudeAdapterV2 background wake turns", () => {
events,
terminalReceipts,
subagentReceipts,
+ providerThreadReceipts,
systemNoticeReceipts,
getOpenedOptions: () => openedOptions,
terminalEvents,
@@ -10092,6 +10100,109 @@ describe("ClaudeAdapterV2 background wake turns", () => {
),
);
+ it.effect("settles an idle-launched subagent discovered during a later user turn", () =>
+ Effect.scoped(
+ Effect.gen(function* () {
+ const harness = yield* makeWakeHarness;
+ const taskId = "task-idle-launch-late-discovery";
+ const toolUseId = "toolu-idle-launch-late-discovery";
+ const startTurn = (ordinal: number) =>
+ harness.runtime.startTurn(
+ makeClaudeTestTurnInput({
+ threadId: harness.threadId,
+ providerThread: harness.providerThread,
+ now: DateTime.makeUnsafe(0),
+ attemptId: RunAttemptId.make(`attempt-idle-discovery-${ordinal}`),
+ providerTurnOrdinal: ordinal,
+ text: "Continue work.",
+ attachments: [],
+ }),
+ );
+ yield* startTurn(1);
+ yield* harness.offerAndWait(
+ makeResultFrame({
+ uuid: "00000000-0000-4000-8000-000000004101",
+ result: "Initial turn finished.",
+ }),
+ );
+ yield* Queue.take(harness.terminalReceipts);
+ yield* harness.offerAndWait(
+ makeSubagentTaskStartedFrame({
+ taskId,
+ toolUseId,
+ uuid: "00000000-0000-4000-8000-000000004102",
+ }),
+ );
+ yield* startTurn(2);
+ yield* harness.offerAndWait(
+ claudeSdkFrame({
+ type: "system",
+ subtype: "task_progress",
+ task_id: taskId,
+ tool_use_id: toolUseId,
+ description: "Checking the result.",
+ uuid: "00000000-0000-4000-8000-000000004103",
+ session_id: WAKE_NATIVE_SESSION,
+ }),
+ );
+ yield* harness.offerAndWait(
+ makeResultFrame({
+ uuid: "00000000-0000-4000-8000-000000004104",
+ result: "User turn finished.",
+ }),
+ );
+ yield* Queue.take(harness.terminalReceipts);
+ yield* harness.offerAndWait(
+ makeSubagentNotificationFrame({
+ taskId,
+ toolUseId,
+ summary: "Audit finished.",
+ uuid: "00000000-0000-4000-8000-000000004105",
+ }),
+ );
+ const markerId = "task-idle-discovery-collector-barrier";
+ yield* harness.offerAndWait(
+ claudeSdkFrame({
+ type: "system",
+ subtype: "task_started",
+ task_id: markerId,
+ tool_use_id: `toolu-${markerId}`,
+ description: "Outcome collector barrier",
+ is_backgrounded: true,
+ task_type: "local_bash",
+ uuid: "00000000-0000-4000-8000-000000004106",
+ session_id: WAKE_NATIVE_SESSION,
+ }),
+ );
+ // A later roster receipt also proves the collector has seen the outcome.
+ yield* Stream.fromQueue(harness.providerThreadReceipts).pipe(
+ Stream.filter(
+ (event) =>
+ event.providerThread.pendingBackgroundTasks?.some(
+ (task) => task.taskId === markerId,
+ ) === true,
+ ),
+ Stream.take(1),
+ Stream.runDrain,
+ );
+ const latest = harness.events.findLast(
+ (event) =>
+ event.type === "subagent.updated" && event.subagent.nativeTaskRef?.nativeId === taskId,
+ );
+ assert.equal(latest?.type, "subagent.updated");
+ if (latest?.type !== "subagent.updated") return;
+ assert.equal(latest.subagent.status, "completed");
+ assert.equal(latest.subagent.result, "Audit finished.");
+ assert.equal(latest.subagent.title, "Audit recent commits");
+ assert.equal(latest.subagent.prompt, "Audit the last five commits.");
+ }).pipe(
+ Effect.provide(
+ Layer.mergeAll(IdAllocator.layer, McpProviderSessions.layer, NodeServices.layer),
+ ),
+ ),
+ ),
+ );
+
it.effect("holds subagent frames that precede task_started and shows its result once", () =>
Effect.scoped(
Effect.gen(function* () {
Activity
juliusmarminge commentedon Oct 20, 2026
Note
Grok responding on behalf of Julius.
Thanks for the deterministic replay. The guard does what you describe on current main. When a
task_notificationarrives while the root is idle, the outcome is applied directly only if notask_startedfor that task is still in the wake buffer, or if a stop outcome is already known (apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts:7112-7125). Otherwise it waits for the continuation drain. #18778 added that deferral so an old buffered start replaying later couldn't flip a finished subagent back to running.The replay depends on step 2, though: it leaves the idle launch's continuation undrained. A subagent
task_startedthat arrives while the root is idle is buffered specifically so that a continuation drains it (lines 6669-6675). In the normal flow that drain processes both the start and the later notification and settles the task. So the open question is what kept that start undrained in your real incident. A held or jammed continuation would make this the same root cause as #20430, or the wake that never opens in #19196, rather than a separate problem with the guard.To narrow it down, could you check the provider log and run history from the production incident:
task_startedand your next user turn?task_notificationarrived?royalaid commentedon Oct 20, 2026
I checked the production run/event history and reproduced this with a fresh native Claude Agent task on the installed macOS nightly
0.1.1-nightly.20261018.3331.Fresh synthetic probe, relative to its idle launch:
task_notification completedarrives while idlerunning,completed_at = nullThere was no T3 continuation/wake run between the idle start and that later message turn, and no queued or held run was found in the durable history. The probe used sequential foreground echo calls, with no SendMessage resume, own background shell work, or interim completion notification. Its last persisted update precedes the final notification by about 68 seconds.
For the three original agents:
The store does not expose the adapter's in-memory
requestedContinuationsset, so I cannot rule out a stuck latch solely from that history. I can confirm that the native Claude wake activity itself occurred without a corresponding T3 run.I agree that removing the guard is wrong: that candidate broke the existing genuine-resume regression and was rejected. The revised candidate keeps the guard and consumes the matching buffered plain-agent start through normal registration before later live progress creates a fallback entry. The ordered adapter replay, real SQLite persistence case, and existing genuine-resume case pass. The broader focused run has 304 passes and one unchanged steer-test timeout, now being checked against baseline. Independent review is running; I will link the patch and installed after-evidence when verified.
royalaid commentedon Oct 20, 2026
The fix is offered in #22049 (through
1e0716cadf). Native fork artifacts are published in preview 20261020.2, commit2ef9c7940e14.The single replay runner runs the same idle-start regression on two prepared worktrees. Apply the test-only patch in this issue to the baseline worktree, install dependencies in both, and run:
It temporarily selects the installed nightly adapter source in the baseline worktree and restores it in
finally. Observed before: exit 1,expected 'running' to equal 'completed'. Observed after: exit 0, the same test passes. This is SDK replay evidence, not a live provider test.Separately, a native synthetic after-probe on the initial installed preview exercised idle self-resume → later-turn discovery → final idle completion:
The old nightly's native probe instead remained running with completed_at null after its final notification. The after-probe used an idle self-resume because notification-woken turns now opened T3 runs after restart; the reason for that wake difference is unproven. Historical stale rows were separately cancelled by normal startup recovery, not retroactively confirmed completed.
The follow-up also covers both active-turn pending frames and idle wake-buffer child frames. Its regression proves idle/live output order, tool completion before the terminal update, sibling isolation and no duplicate output on a later continuation drain. 53 focused checks, server typecheck and native Windows/macOS package validation pass. Independent review found no material defects. The unrelated early interim-completion behavior (#16610) remains.
royalaid commentedon Oct 22, 2026
You were right to ask about held runs. Our earlier answer to (2) only looked at runs 14 to 28, and it was wrong. I've now reproduced the whole route on current main through the app, with no injected frames.
Repro on main 3b95e7831b, Windows, web dev client, Claude Sonnet 5.5
Every message after the first Stop goes through the MCP /
t3 thread sendpath, so none of them is a client message that would release the hold (#21473).echocalls. While it runs, send a follow-up so it queues. Press Stop in the composer. Run 1 is interrupted and the queued run 2 becomesqueued,queueHeld: true.t3 thread send): startnode -e "setTimeout(()=>{},30000)"withrun_in_background: true, then run a 26 s foregroundnode -e "setTimeout(()=>{},26000)", then write a ~300-word reply and end the turn. When the background command's notification arrives, call Agent withrun_in_background: true(60 sequentialecho probe3-step-ncalls), then end that turn. The point of the timing is that the background command finishes while the run is still settling.t3 thread send.claude-wake-turn-detected; the continuation is dispatched as run 7, createdqueued,queueHeld: true,queuePosition: 2, while run 6 is stillwaitingAgent "lineage probe 3" finishedwith the parent idlerunning,completed_atnull,updated_at05:44:08Two controls from the same thread:
The original incident, on
0.1.1-nightly.20261018.3331The same shape occurred in normal use.
thread.runtime-mode.setdetached the session, run 1 failed, and the already queued run 2 (a delegated-task wake) becamequeueHeld: true.queuePosition: 4. None of them started.Conclusion
The stale row follows from the open latch half of #20430. The held continuation keeps
requestedContinuationsset, so later wake turns, and the Agent launches inside them, never get a run. The start stays buffered, and the final notification waits behind it. Theo asked on #20430 for a separate issue covering only the latch, so I've retitled this one to it. With the latch fixed, wake turns would drain the start normally. #22049 then only matters if a start can be stranded some other way, and we're fine closing it if you'd rather not carry it as defense in depth.Sanitized excerpt of the original probe window (3331)
Ids, hosts, paths and names are removed, and external message bodies are summarised. "T3 run" means a row in
orchestration_v2_projection_runswhose window contains the turn start. All times are UTC on 20 Oct.run_in_background: true, threesleep 40; echosteps)<task-notification>: probe A "is waiting on its own background work" (marked completed early, #16610)sleepwas blocked); parent launches probe B2 at 00:20:44.599running00:21:08.911 to 00:21:13.639Agent "lineage probe B2" finished,<result>PROBE_B_OK</result>B2's Agent input:
{"description": "lineage probe B2", "subagent_type": "<custom haiku agent>", "run_in_background": true, "prompt": "Synthetic test task with no real work. Make 60 separate Bash tool calls, strictly one after another and never in parallel, numbered n = 1 to 60. Call n runs exactly `echo probe-step-n` with the number filled in. Do not use run_in_background, sleep, Monitor or any loop: one echo per tool call. After all 60, reply with exactly: PROBE_B_OK"}The projection stayed
runningwithcompleted_atnull until startup reconcile cancelled it at 00:54:29.589. Runs 2, 8 and 12 werequeuedwithqueueHeld: truethroughout.[-]Claude idle-launch completion is blocked by an older buffered start after task_progress discovery[/-][+]Claude wake latch: a wake queued behind a held run swallows later wakes, so idle-launched subagents stay Running[/+]royalaid commentedon Oct 22, 2026
Follow-up: I reran the same route against both open fixes, on main 3b95e7831b with the same Stop, MCP-only messages and timing as above.
#22049 alone does not cover it. The wake continuation (run 4) was again created
queued,queueHeld: truewhile run 3 was settling, and never started. The Agent launched in the untracked wake turn at 07:51:50Z. The later ping (run 5, 07:52:54 to 07:53:00Z) happened to fall between two of the agent'stask_progressframes (07:52:53.8Z and 07:53:00.2Z), so the buffered-start replay never fired. The agent finished at 07:55:29Z with the parent idle, and T3 never recorded it at all: no subagent row, run 4 still held. #22049 only helps when a progress frame happens to land inside some later run.#17885 does cover it. With its Orchestrator and ProjectionStore changes applied on the same base (only conflict was the Codex test assertion, since that test is gone on main):
waiting; background command completes, wake dispatchedrunningcompletedFocused suites pass on that build: runtimeLayer, ProjectionStore, ThreadLaunchService, BackgroundWorkStop, ThreadStop and CodexAdapterV2 (286 tests). That build also carried #22049, but it wasn't exercised: the start arrived inside run 4.
So for this route the fix is #17885, which needs a rebase onto main. What it leaves open, by its own description, is a wake dispatched after Stop reaches the active run: that one is still held, and the latch stays set until the CLI restarts. #17109's release-on-cancel/fail would close that. I'm happy to close #22049 in favour of #17885.