Repository navigation
[Bug]: A busy orchestrator thread's queue grows without bound: PR-watch wakes and recurring tasks never coalesce (75 queued, 59 stale PR wakes) #26987
Description
Note
🤖 Claude Opus 5.5, writing on behalf of the reporter.
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
This is related to three open issues but is not a duplicate of any of them. It is what happens when all three hit one long-lived thread at the same moment:
- [Bug]: A thread watching several pull requests gets one wake and one turn per pull request from the same pass #19183: one wake per pull request from the same pass;
- Queued PR-watch wakeups survive unwatch, merge, and a newer head #21009: queued wakes survive a newer head;
- [Bug]: Recurring automations accumulate queued prompts while threads are blocked #16818: recurring automations accumulate queued prompts.
Area
apps/server
Summary
We run one long-lived orchestrator thread. It watches the PRs its child workers open, and two recurring scheduled tasks are bound to it: a sweep every 30 min and a check every hour. Over about a day the thread's queue grew to 75 undelivered messages, and new user messages waited behind all of them. The thread spent its turns on notifications that were already stale. Meanwhile, workers that had finished sat idle because the orchestrator never reached them. Only 2 of 10 worker slots were in use.
What was in the queue (snapshot)
| Kind | Queued |
|---|---|
Update on pull request #N … PR-watch wakes |
59 |
| Copies of the same 30-min recurring sweep prompt | 6 |
| Copies of the same hourly recurring prompt | 3 |
| Real messages from users or other threads | 7 |
The 59 PR-watch wakes covered only 13 PRs. Per PR: 9, 9, 8, 6, 6, 5, 4, 3, 2, 2, 2, 2, 1. Most were superseded before they could be delivered. A typical sequence for one PR was "Checks failed on A", then "3 new comments", then "Checks failed on B", then "All checks passed on C". All four stayed queued, and each would run as its own turn.
Thread size when we retired it: about 35,000 timeline items over 3,866 runs in 8 days. Each turn on a thread that size is slow. That slows draining, the queue grows faster than it drains, and the backlog snowballs.
Steps to reproduce
- Have one thread call
watch_pull_requeston about 10 active PRs. Child workers push to them several times an hour, and review bots comment. - Bind a recurring scheduled task with an interval of 30 min or less to the same thread.
- Keep the thread busy, so that each turn takes longer than the gap between wakes. A large thread does this naturally.
- Inspect the thread's queue (
t3_queuelist) after a few hours.
Expected behavior
- PR-watch wakes coalesce per thread and per PR while undelivered. A newer update for the same PR replaces or merges into the queued one: latest head, latest check state, and a count of new comments. That covers Queued PR-watch wakeups survive unwatch, merge, and a newer head #21009's newer-head case and also comment-only updates.
- One pass across many PRs gives one wake ([Bug]: A thread watching several pull requests gets one wake and one turn per pull request from the same pass #19183).
- A recurring task skips or coalesces when its previous occurrence is still queued on the target thread ([Bug]: Recurring automations accumulate queued prompts while threads are blocked #16818).
- A user's own message is not starved behind automatic wakes. For example, user messages could be ordered ahead of automatic deliveries; see also t3_queue_reorder can't move a message ahead of queued automatic completion deliveries #25646.
Actual behavior
Every update for a PR, and every recurring tick, is queued as its own message and runs as its own turn. Nothing invalidates or merges the queued copies. The queue is unbounded.
Cancelling the backlog by hand took 66 separate t3_queue cancel calls, about 7 minutes in total. Some sends and cancels to that thread returned HTTP 503 while it was in this state.
Impact
Major degradation or frequent failure. The orchestrator stopped responding in practice, and user messages appeared to be "swallowed". The workaround was to cancel the queue by hand, retire the thread and start a successor.
Version or commit
T3 Code (Beta) 0.1.5, macOS 15 (Darwin 24.1), Codex provider on the orchestrator thread. statev2.sqlite was 16 GB at the time, which may be a factor in the slow turns and 503s (#24895, #21840).
Workaround we're using
The orchestrator no longer watches PRs; the worker that owns a PR watches it. Another thread trims stale wakes from the orchestrator's queue every hour, and the orchestrator is handed off at about 12k items.
Activity
onlydole commentedon Nov 3, 2026
Two single-PR cases where the wakes didn't coalesce, both on GitHub repos with review bots.
In the second case, Codex also called
link_pull_requeston the same PR 3 times (19:59, 20:02, 20:08), even though each update says "This PR stays linked and watched, so do not link, watch, or list it again."genesiscz commentedon Nov 5, 2026
A data point for the coalescing side, on T3 Code 0.1.7 (macOS arm64 desktop, Claude provider). The unwatch side of the same case is in #21009 (comment).
One busy orchestrator thread watched about 25 PRs. In one day it had 466 PR-watch runs: 350 completed (350 full-context turns woken by PR notices) and 115 still queued after every watch had ended.
No coalescing. When the 115 were removed, 15 PRs had between 1 and 13 queued notices each: 13, 12, 12, 10, 10, 10, 10, 9, 8, 8, 4, 3, 3, 2, 1. On
main(337c5858c) each sweep that finds a change dispatches a sync with a freshmessage:pr-watch:<uuid>(PullRequestWatchReactor.ts:279-290), and the orchestrator turns it into a newqueue_after_activemessage (Orchestrator.ts:3025-3040). Nothing looks for a queued notice for the same PR first.No bulk cleanup. Removing them took 115
t3_queuecancelcalls over about 4.5 minutes (66 in the original report).canceltakes onequeuedRunId(thread/tools.ts:83-109) and dispatches onequeued-run.cancelwith onerunId(orchestrationV2.ts:3259-3264). The only path that cancels all automatic queued runs together isthread.settle(Orchestrator.ts:3373-3384), which refuses while a run is active (Orchestrator.ts:3399) and settles the thread, so a busy orchestrator cannot use it.Suggestions, in addition to the ones above:
t3_queuecancel, for examplequeuedRunIds: [...]or a filter such assource: "monitor". It is small, and it is the escape hatch for any automatic source that floods a busy thread.Written by Claude Opus 5.5 in T3 Code for the reporter, from read-only queries on the local database.