Skip to content

[Bug]: Original delegate_task task finalizes on a stale result in the gap before the child's background-exit wake turn #24297

Open

Description

@CookerTea

Before submitting

Area

apps/server

Steps to reproduce

  1. A parent thread starts a Claude child with delegate_task (mode: "async").
  2. The child starts a long Bash command with run_in_background and also starts two reviewers of its own with delegate_task, then ends its first turn ("still running, waiting on the checks").
  3. The task correctly sits in waiting_for_children.
  4. The background command exits. Claude Code starts a wake turn in the child for that exit.

Expected behavior

The task stays open until the child's wake turn (and the turns started by its reviewers finishing) have run, and the parent receives the child's last result.

Actual behavior

The task finalizes in the gap between the background command exiting and the wake turn being registered, using the stale text from run 1 as the result. Timestamps from the state DB (statev2.sqlite), one case:

22:50:35.143  task finalized, summary = run 1's "still running" text
22:50:35.251  parent woken with that summary
22:50:35.330  child run 3 starts (the wake turn for the background command's exit)
later         runs 4 and 5 start when the child's two reviewers finish; run 5 has the real final result

A finalized task never reopens, so runs 3–5 belong to no task and their result is never delivered to the parent. task_status shows latestTerminalSummary for the later runs, but nothing wakes the parent.

It looks like delegatedTaskProgress sees no non-terminal run and no pending background item for a moment (the command has ended, the wake run isn't created yet) and finalizeAppOwnedSubagent closes the task on that snapshot.

Impact

Major degradation or frequent failure: 3 missed results in 2 days across our parent threads (this race once, the #21837 follow-up path twice).

Version or commit

T3 Code 0.1.2 (stable)

Environment

Ubuntu 24.04 server, Claude provider (Claude Code SDK), claude-opus-5-5

Workaround

Children never end a turn while a background command or reviewer they started is still running. They wait inside the turn (t3_thread_wait on reviewer threads, foreground waits for commands), and a Claude Code Stop hook blocks a child's turn end without a final status line. Parents read task_status before trusting a wake.

Activity

  1. dilbagh commented on Oct 25, 2026

    @dilbagh

    Note

    Claude Opus 5.5 posting on behalf of @dilbagh

    Two more hits of this race in one parent thread on 0.1.2 stable (Ubuntu 24.04, Claude Code 2.1.309; parent Claude Opus 5.5, children Claude Fable 5.1, all delegate_task async).

    Each child opened a PR, started gh pr checks --watch with run_in_background, and ended its turn saying it was waiting on CI. When CI finished, the parent got the "Delegated task … finished" notification a few tens of milliseconds before the child's wake run was requested:

    Child Parent notified Child wake run requested Gap Child's final report
    A 01:21:55.267 01:21:55.324 57 ms 01:22:20
    B 04:41:18.351 04:41:18.390 39 ms 04:41:47

    In A's case task_status at notification time showed hasPendingChildRuns: true and latestTerminalSummary: null, but the task was already completed / result_available. The parent took the stale "waiting on CI" text as the state and ended its turn. The child's later final report ("CI green, PR Ready") showed up in task_status, but nothing woke the parent. Both PRs sat ready to merge until the user asked for a status, about 2 h and 1 h 40 min later.

    The workaround we're using: children stop at a Ready PR without waiting on CI, and the parent calls watch_pull_request itself. That avoids the child's background-exit wake altogether.

  2. juliusmarminge commented on Oct 26, 2026

    @juliusmarminge
    Member

    Same race, seen twice on 2026-10-26 with a Claude child whose background work is native Agent subagents (run_in_background), not a Bash command. Nucbox-1, 0.1.3-preview.20261026.3702, parent thread 9b9ea64f-ecdf-4b2e-85aa-5346ab2a0197.

    Task fleet-audit-vm-leads-20261026: the child starts three background readers (A, B, C) in run 1.

    UTC
    20:49:47 run 1 ends ("The three transcript readers are still running")
    20:50:08 / 20:50:36 readers B, C finish, wake runs 2 and 3
    20:50:46 run 3 ends: "Reader C is back… Waiting on reader A"
    20:51:24.142 reader A finishes
    20:51:25.645 task finalizes: subagent_result transfer from run 3 to the parent (run 168)
    20:51:29.937 wake run 4 requested
    20:51:34 parent receives … (Audit: VM lead agents) [completed]: Reader C is back (PR leads, Codex vs Claude). Waiting on reader A: …
    20:54:32 run 4 writes the real result. It arrives as a second delivery under a different task id (provider-continuation…).

    Task fleet-audit-pr-dispatcher-20261026: same pattern. The last reader finishes at 20:58:33.216, the task finalizes from run 3 ("One reader … is still running") at 20:58:37.024, and wake run 4 is requested at 20:58:42.898. The parent got the interim text as [completed] at 20:58:44.

    Both times the finalization lands in the 1.5–4 s gap between the native subagent finishing and the wake run being registered, the same gap as above. hasPendingChildRuns (apps/server/src/mcp/OrchestratorMcpService.ts:472, origin/main) only counts later runs of the child thread, so neither a running native subagent nor a not-yet-registered wake run keeps the task open.

  3. DanielJomphe commented on Oct 27, 2026

    @DanielJomphe

    Simpler repro than the one above: the child doesn't need to start reviewers of its own; one background Bash command is enough. Call delegate_task with mode: "async" on a Claude child with this task, then let the notices arrive without calling task_status.

    1. Run this in one Bash call with run_in_background set to true:
       sleep 75 && date -u +%H:%M:%S > /tmp/t3-wake-repro-1.txt
    2. Do not wait for it. Reply with exactly "REPRO1 WAITING" and end your turn.
    3. When you are woken because that command finished, read the file and reply "REPRO1 DONE <its contents>".
    

    On build 0.1.4-nightly.20261027.3770, with a Haiku child, the task completed with REPRO1 WAITING 0.02 s after the command exited. REPRO1 DONE … arrived 5 s later as a second task with the same title, through #23542's follow-up path. The timings are from 3770; 3783 has the same delivery code.

    #24391 on current main. Its ClaudeAdapterV2.ts changes no longer apply cleanly at 570d4b03ee: claudePendingBackgroundTasksFromRoster now takes awaitedTaskIds, and the idle emit moved into emitRosterUpdate. A port is below. A single-ordering version of #24391's new test fails on main with expected [ [] ] to deeply equal [ [ 'bdqirlcyw' ] ], and passes with the port.

    A newer test conflicts with it. "holds only the background tasks the agent marks awaited, until they end" landed after #24391's base, and times out with the port. While the thread is idle, a snapshot drops one of two tasks without a task_notification, and the test waits for the published roster to shrink to one entry. The port keeps the dropped task published as owed a wake, so it never shrinks. Either the test or that rule needs to change. #24391 already rewrote the older "uses task_started as an incremental roster fallback…" test, where an empty snapshot drops a task the same way.

    #24391 ported to main at 570d4b03ee (ClaudeAdapterV2.ts only)
    diff --git a/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts b/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts
    --- a/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts
    +++ b/apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts
    @@ -4972,6 +4972,7 @@ export const makeClaudeAdapterV2 = Effect.fn("makeClaudeAdapterV2")(function* (
               readonly nativeThreadId: string;
               readonly providerThread: OrchestrationV2ProviderThread;
               readonly status?: OrchestrationV2ProviderThread["status"];
    +          readonly owedWakeTasks?: ReadonlyArray<OrchestrationV2PendingBackgroundTask>;
             }) {
               const roster = rosterForNativeThread(
                 yield* Ref.get(pendingBackgroundTasksByNativeThread),
    @@ -4986,7 +4987,10 @@ export const makeClaudeAdapterV2 = Effect.fn("makeClaudeAdapterV2")(function* (
                 ...input.providerThread,
                 providerSessionId: session.id,
                 ...(input.status === undefined ? {} : { status: input.status }),
    -            pendingBackgroundTasks: claudePendingBackgroundTasksFromRoster(roster, awaitedTaskIds),
    +            pendingBackgroundTasks: [
    +              ...claudePendingBackgroundTasksFromRoster(roster, awaitedTaskIds),
    +              ...(input.owedWakeTasks ?? []).filter((task) => !roster.has(task.taskId)),
    +            ],
                 ...(goalsByNativeThread.has(input.nativeThreadId)
                   ? { goal: goalsByNativeThread.get(input.nativeThreadId) ?? null }
                   : {}),
    @@ -7692,13 +7696,30 @@ export const makeClaudeAdapterV2 = Effect.fn("makeClaudeAdapterV2")(function* (
               if (baseThread === undefined) {
                 return;
               }
    +          // While idle, a published task Claude still owes a wake run stays published after the
    +          // edge or level drops it (their order is unspecified): an empty roster lets a delegated
    +          // child finalize on its pre-wake result. Only the published roster keeps it, so it never
    +          // blocks query replacement; the wake run's finalize or a process reset publishes it away.
    +          const owedWakeTasks: OrchestrationV2PendingBackgroundTask[] = [];
    +          if (activeContext === null) {
    +            for (const task of baseThread.pendingBackgroundTasks ?? []) {
    +              if (yield* isKnownOpaqueBackgroundTaskOnNativeThread(nativeThreadId, task.taskId)) {
    +                owedWakeTasks.push(task);
    +              }
    +            }
    +          }
               const status =
                 activeContext === null
                   ? ("idle" as const)
                   : baseThread.status === "idle"
                     ? ("active" as const)
                     : baseThread.status;
    -          yield* emitProviderThreadRoster({ nativeThreadId, providerThread: baseThread, status });
    +          yield* emitProviderThreadRoster({
    +            nativeThreadId,
    +            providerThread: baseThread,
    +            status,
    +            owedWakeTasks,
    +          });
             });
     
             const applyBackgroundTaskRosterMessage = Effect.fnUntraced(function* (input: {

    Reported by DanielJomphe. Investigation and tests by Claude Opus 5.5 in Claude Code, through T3 Code.

  4. Hassan-Zbib commented on Oct 29, 2026

    @Hassan-Zbib

    Note

    Claude Opus 5.5 (Claude Code) posting on behalf of @Hassan-Zbib.

    Still happening on 0.1.4-nightly.20261028.3814 (Windows 11, Claude Code 2.1.312, Opus 5.5 parent and child, async delegate_task). That build has #24610 and #23542 but not #24391. A t3_background_wait mark doesn't close the gap.

    The child started clippy with run_in_background as task b3gyl80d9. It called t3_background_wait({"taskIds":["b3gyl80d9"],"waiting":true}), which matched, and ended run 1 with "Waiting on clippy…". Timeline from statev2.sqlite, UTC:

    19:21:55.851 Child run 1 ends; the mark holds the task open
    19:23:09.382 Original task completed with run 1's "Waiting on clippy…" as its result
    19:23:09.451 Wake run 2 for the command's exit is requested, 69 ms later
    19:23:09.523 Parent woken with the interim text

    The mark holds the task until the command exits, then the task finalizes in the gap before the wake run registers. The real result still arrived later through #23542's per-run follow-up tasks. While it was pending, another child's task_status read completed / result_available with hasPendingChildRuns: true. Those follow-ups completing early is #21837's path; I've commented there.

  5. mkiera commented on Oct 30, 2026

    @mkiera
    Contributor

    Note

    Comment left by Claude Opus 5.5 (Claude Code) on behalf of mkiera.

    Still happening on 0.1.4-nightly.20261029.3853 (Windows 11, Claude Opus 5.5 parent and child, async delegate_task).

    The child started a dotnet publish with run_in_background, called t3_background_wait (matched) and ended run 1 at 11:30:20 local. The command exited and the task finalized at 11:33:13 with run 1's text. Run 2 (the wake turn) started at 11:33:14.

    Run 2 started a new background publish, marked it with t3_background_wait (matched at 11:33:33) and ended at 11:33:35. Its follow-up task finalized at once, while that command still ran. Run 3 repeated this at 11:38:00.

    Each finalize told the parent "finished" while the build was still running.

  6. saphitv commented on Oct 31, 2026

    @saphitv

    Note

    🤖 gpt-6.1-sol responding on behalf of saphitv

    We encountered the same premature-completion symptom today with linked-environment delegate_task and a Claude Opus 5.5 child. The parent marked the task completed using an interim “capture is running in the background” reply, while the child continued through three more automatic runs. The parent card retained that first reply.

    The child successfully registered its background Bash task with t3_background_wait before ending its first turn. Incident timeline on October 31, 2026 (UTC), from thread activity and read-only run records:

    11:09:42.357  t3_background_wait returns matched=[background task ID]
    11:09:44.825  Child run 1 completes with an explicitly interim reply
    11:10:22.945  Child run 2 requested automatically
    11:10:22.994  Parent task card updated to completed with run 1's reply
    11:10:23.044  Parent receives “Delegated task … finished”
    11:12:14.837  Child run 3 requested
    11:14:38.778  Child run 4 requested
    11:15:26.248  Parent's thread-read response shows child still running run 4
    11:17:57.594  Child run 4 completes
    

    Affected desktop client and destination server: 0.1.6-nightly.20261031.3977 (2359643cffa6), macOS arm64, Node v24.22.1. The parent was on a linked Linux server whose version was not established.

    The early completion and stale result are confirmed; attribution to this particular race remains an inference. No controlled reproduction or origin-side event trace was collected. This adds a remote-delegation case to the symptoms reported here.

    Workaround for this incident: inspect the destination thread's active run and actual final reply directly, rather than trusting the origin's completed card. No runtime changes were made during triage.

  7. boldprojekte commented on Nov 1, 2026

    @boldprojekte

    Another occurrence of the same race, Claude child (claude-opus-5-5) on T3 Code with an async delegate_task:

    01:14:04.0   child starts a long check in the background (Bash run_in_background)
    01:16:19.6   child ends run 1: "The check is still running. I'll be called again when it finishes."
                 task stays open, parent is not woken (correct)
    01:22:04.9   background command exits
    01:22:07.415 child run 2 starts, notification source background_command "finished"
    01:22:07.620 parent woken: Delegated task "…" finished, summary = run 1's stale text
    01:23:06.8   child run 2 completes with the real result
    

    The parent read the stale summary, concluded the child had been abandoned and sent it a follow-up. The real result reached the parent only because that t3_thread_send opened a follow-up task (finished 01:23:08). Without it, run 2's result would have been lost. Timestamps from statev2.sqlite (orchestration_v2_projection_runs, …_turn_items).

  8. mindyk commented on Nov 9, 2026

    @mindyk

    Note

    🤖 Claude Opus 5.5 responding on behalf of mindyk

    Still reproduces on v0.1.7, and calling t3_background_wait does not prevent it. Main at 4df0c130 has the same logic.

    Minimal repro (Claude parent, claude-haiku-5-5 child, delegate_task async with shared isolation). The child's brief:

    1. Start sleep 130 as a tracked background Bash task, call t3_background_wait with its ID (returns matched), and end the turn with "INTERIM 1".
    2. On the task notification, do the same with a second job and end with "INTERIM 2".
    3. On the next notification, end with "FINAL".

    The parent receives three completions:

    • "INTERIM 1" under the original task ID. It arrives when job 1 exits, before the child's wake turn runs. This is the race this issue describes.
    • "INTERIM 2" under a provider-continuation: task ID. It arrives about 1 s after that turn ends, while job 2 is still awaited.
    • "FINAL" under another continuation ID.

    The second delivery looks like the follow-up gap in #21837 / #16603. In finalizeAppOwnedSubagent, the progress.state !== "result_available" hold applies only when task.id === childControls.thread.forkedFrom.nodeId, so continuation tasks are finalized when their run ends, even with awaited background work pending.

    A production squad worker hit the same thing: three 18–27 min benchmark series, each started with run_in_background plus t3_background_wait, produced three premature completions.

    The workaround we use: the child never ends its turn while its work runs. It waits in foreground chunks of timeout 540 tail --pid=PID -f /dev/null. With that, the same probe delivers exactly one completion.

  9. 1kuna commented on Nov 9, 2026

    @1kuna
    Contributor

    Title: Historical frequency evidence for stale delegated-task completion notices

    Problem: In a read-only audit of 4,846 finished app-owned delegated tasks on one T3 Code 0.1.6 installation (macOS arm64), 93 tasks (1.9%) closed within 3 seconds of a new child run starting. In 74 of those cases, the parent notice was byte-identical to the previous turn's interim text. The later result was not delivered to the parent.

    Reproduction or observation: These are retained database observations, not a fresh reproduction on current main. The near-finalization child runs were triggered by background-command exits (62), peer messages (29), and grandchild notices (2), across Claude and Codex children. The follow-up turn took a median 54 seconds (p90 9 minutes). In hand-checked cases, parents repeated the stale "still running" state even after the real result existed.

    The first completion path was generally prompt: 4,780 of the 4,846 tasks reached a parent within 10 minutes through a new run, steering into an active run, or return to a still-live blocking wait. That aggregate does not establish that every notice carried the final result.

    Proposed behavior: Keep a task pending while the child owes a wake turn, and deliver the result of that turn rather than the previous interim message. If the task has already closed, deliver subsequent output once and make the latest result discoverable. #24391 remains open at this check; this is additional historical frequency evidence for its direction, not a claim that its patch was tested here. Later-turn delivery is also tracked in #28448.

    Why native: T3 owns task finalization and provider wake registration. A parent relying on completion delivery cannot repair the race from a stale completion summary without separately inspecting the child.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions