Repository navigation
CI: hosted runner queue saturates repo-wide (~425 queued runs), so PR checks wait hours behind push and workflow_run traffic #2605
Description
The hosted runner queue in this repository is saturated for long stretches, and every worker shift that waits on CI burns its timebox and parks. This issue records the measurements so the pattern does not have to be rediscovered each shift.
Observed on 2026-10-30 (queue worker repo-worker-code, several consecutive shifts):
- 425 workflow runs queued, 5 in progress repo-wide at 07:03Z.
- Of the 100 most recent queued runs at that moment:
pull_request=40,push=23,workflow_run=37. - A
pull_requestrun for PR feat(mcpx): single-source the version in pkgs/mcpx/VERSION (#1881 slice 2) #2572 has beenqueuedsince 05:18Z (headb118a09f); PR feat(mcpx): release workflow + curl|sh installer + container publication (#1881 slices 4-6) #2584's since 06:12Z. Neither had a runner assigned ~2 hours later. - Push-triggered runs do get picked up:
main broken watchercompletes within seconds, so runners exist — the queue is simply far deeper than the runner pool, and PR runs sit behind push/workflow_run traffic. - Consequence for the queue: tasks whose only remaining step is "merge when checks are green" park repeatedly (task 3114 has parked three shifts running), because a 45-minute timebox cannot outlast a two-hour queue.
Likely contributors worth checking (not verified):
- The number of workflows per push/PR in this repo (
gh workflow list | wc -lis large) multiplied by concurrent agent branches. workflow_run-triggered chains (37 of the sampled 100) re-enqueueing each other.- Whether the repo is on a shared/free runner pool with a concurrency ceiling.
Possible directions, in likely order of payoff: scope heavy workflows to paths: so unrelated PRs do not queue them; consolidate the per-package go build matrix jobs into one job; give PR workflows a higher effective priority by cancelling stale in-flight runs of the same ref (concurrency: cancel-in-progress: true where safe); and consider whether the fleet should cap concurrent agent PRs so the queue can drain.
Filed from queue task 3114, which is blocked on exactly this (PRs #2572 and #2584, both MERGEABLE, waiting on green checks).
Acceptance criteria
- The dominant queue contributors are measured (workflow count per push/PR, workflow_run chain fan-out, stale same-ref runs) and the largest one is named with numbers
- At least one structural reduction lands (paths: scoping, matrix consolidation, or concurrency cancel-in-progress for stale same-ref PR runs) via branch + PR, or is explicitly deferred with the reason recorded here
- A post-change measurement shows PR checks' median queue wait materially reduced from the ~2h observed on PRs feat(mcpx): single-source the version in pkgs/mcpx/VERSION (#1881 slice 2) #2572/feat(mcpx): release workflow + curl|sh installer + container publication (#1881 slices 4-6) #2584
Activity
dezren39 commentedon Oct 30, 2026
Intake review
Related: Tasks repeatedly parked on this (e.g. task 3114 / PRs #2572 and #2584, named in the body); #1599 (earlier CI runner stall tracking)
Notes: Filed from queue task 3114's shift after direct measurement at 07:03Z: 425 workflow runs queued repo-wide, 5 in progress; of the 100 most recent queued, 40 pull_request / 23 push / 37 workflow_run; PR runs waited ~2h with no runner while push-triggered watcher runs complete in seconds. This pass appends acceptance criteria and applies kind/ci, priority/high (every worker shift burns its timebox against this) and effort/m; the issue already carries status/needs-triage.
Risks: Until the queue drains faster, merge-when-green work parks repeatedly and fleet throughput is capped by runner wait, not by work; aggressive cancellation settings could also kill legitimate long runs if scoped wrong.
Suggested approach: Measure first (per-workflow queue contribution), then apply the cheapest structural cut — likely concurrency cancel-in-progress on PR workflows and paths: scoping for heavy per-package workflows — one workflow family at a time, with before/after queue measurements.
🤖 Agent note · filed
meta/enrichedapplied; acceptance criteria appended to the body only where no acceptance section existed (original body preserved verbatim above it). Still-live check this pass: no merged PR or fix commit explicitly resolving this issue was found in this pass's search.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7345 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
4 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: done (triage) — task 7345 completed by lane code-n, outcome
triaged. Issue #2605 stays open; the structural fix is queued as task 8924 (small_fix) in repo-health-code-pilot, soqueue/queuedreflects that live follow-up. No blockers.dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: handed_off
task: 8924
lane: code-j
worker: repo-worker-code
execution_lane: opencode
issue: #2605
kind: small_fix
state_checked: 2026-11-04T12:00Z — issue #2605 OPEN, no held/user-owned/do-not-merge/do-not-auto-edit labels; labels synced to queue/claimed + status/in-progress + work/assigned.
next: opencode claimant launches the bound session for task 8924.