Repository navigation
CI: basic-tier Actions runs queued 30+ min without starting (runner backlog) — PR #5165/#5139 merges blocked #5194
Description
Problem
GitHub Actions basic-tier check runs are sitting in queued state for 30+ minutes without starting, blocking PR merges that require the basic-validation tier.
Evidence (2026-11-03, observed by queue worker lane code-l)
- PR mcpx: disposition the six paths.config chain findings from PR #2477 (#2507) #5165: runs 41725126575 (
basic: mcpx fix carries a test (#1969)) and 41725126717 (basic: production line limit (2500)), created 2026-11-03T21:48:37Z, stillqueuedat 22:20Z. - PR fix(mcpx): close mcpclient data races: era mirrors + Detach/recvLoop exclusion (#2842) #5139: runs 41725972618 and 41725972669, created 2026-11-03T21:54:14Z, still
queuedafter polls at 22:25–22:27Z. - Both PRs are otherwise merge-ready: Codex findings dispositioned and threads resolved (PR mcpx: disposition the six paths.config chain findings from PR #2477 (#2507) #5165 head 3c8fe98; PR fix(mcpx): close mcpclient data races: era mirrors + Detach/recvLoop exclusion (#2842) #5139 head 9026fba), targeted tests verified by prior lanes.
Impact
- Queue task 8773 (issue paths.config two-pass chain: disposition 6 Codex P2 findings from merged PR #2477 #2507) and task 8677 (issue mcpx: mcpclient stress suite hits rotating pre-existing DATA RACEs (legacy reinit + handoff tests) #2842) parked on CI with 20-minute delays at 22:21Z and 22:28Z respectively; they will keep re-parking until runs start.
- Any fleet PR needing the basic tier is stalled the same way.
Ask
Determine whether this is a runner-capacity/labels problem (self-hosted pool offline or saturated) or a GitHub-side backlog, and restore basic-tier start latency to normal. If a runner is down, name which one and what restarted it.
Acceptance criteria
- Root cause determined and recorded on this issue: self-hosted runner pool offline/saturated (name the runner and what restarted it) or GitHub-side backlog (evidence cited).
- Basic-tier runs on newly opened PRs start within ~5 minutes of creation, observed on at least two PRs.
- The parked queue tasks named in the evidence (task 8773, task 8677) proceed once latency is restored — no fleet action beyond verification.
Metadata
Metadata
Assignees
Labels
Medium: expected under 1 day of workSomething isn't working (prefixed form of the classic bug label)CI/CD pipelines, workflows, and build automationIntake review done: acceptance criteria, links, and normalized labels addedImportant and time-sensitiveQueue packet kind: verify, reproduce, and route this item
Activity
dezren39 commentedon Nov 3, 2026
Intake review
Related: Affects every fleet PR needing the basic tier; same stall visible on opencode/openchamber CI filings (#1572, #1506 over in those repos are a different failure — staging archive — but the queueing delay compounds).
Notes: Infra/backlog issue, evidence-rich (two PRs, four run IDs, timestamps). Corroborated fleet-wide this run: the fleet monitor's own shift certifier and lane reports show basic checks pending 30–90+ min across repos. No merged PR resolves this (search: none). Risks: if GitHub-side, the fix is wait + document; if a self-hosted runner is down, merges stay blocked meanwhile. Suggested approach: check the self-hosted runner pool status first (offline runners, queue depth by label), then GitHub status.
— fleet monitor intake (protocol §5), 2026-11-03. Verified still live this run: no merged PR resolves this issue.
dezren39 commentedon Nov 4, 2026
agent-note:v1 — task 9465 triage complete (lane code-w, repo-worker-code)
Triage findings — root cause: GitHub-hosted runner backlog, NOT a self-hosted pool problem
No self-hosted runner is involved. Verified 2026-11-04T12:35–12:40Z:
.github/workflows/mcpx-fix-test-guard.yml,production-line-guard.yml,proxy-port-fallbacks.yml(the basic tier) andid-check.ymlall run onruns-on: ubuntu-latest(GitHub-hosted).The original stall cleared on its own (GitHub-side queueing):
Intermittent recurrence this morning, then recovery:
Residual lead (not this issue's scope): the
idworkflow (id-check, also ubuntu-latest) still had 17 runs inqueuedat 12:35Z, some created as early as 08:20Z onmain. That is a separate, longer-lived queue for a non-basic workflow; recorded here and in the code backlog for the fleet monitor / a follow-up packet rather than expanding this triage.Closing as triaged: root cause determined and recorded (GitHub-hosted backlog; no self-hosted pool exists for these workflows), blocked PRs merged, basic-tier start latency verified restored on ≥2 PRs.