Repository navigation
ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599
Description
Problem
As of 2026-10-28T05:50Z, GitHub Actions check runs for developing-today/code PRs are stuck in queued and not being picked up by any runner. Zero check runs have failed — they simply never start. This blocks every fleet merge that requires green CI.
Evidence (queue worker lane C shift, 2026-10-28 04:54–05:52Z)
- PR id(directory): drop + truncate torn final journal line in replay (#1521) #1563 @ ec0ce84 (head pushed ~04:44Z): 31 check runs; the 26
nix flake check (*)runs were still all statusqueuedat 05:48Z (>1h), 3 passed earlier, 0 failed. - PR fix(id): widen spaced-destination protection to tab/newline destinations (#1423) #1543 @ 802f154: 8 success / 21 queued / 0 failed at 05:27Z; no movement during a 5-minute poll.
- PR id(web): close image browser on editor teardown (#1381) #1588 @ b4210bd: 3 success / 26 incomplete / 0 failed; no movement during a 4-minute poll.
- PR fix(id): do not claim paste/drop when every image is oversized (#1321) #1576 @ f4d430f: 4 success / 26 incomplete / 0 failed; no movement during a 3-minute poll.
- PR feat(mcpx): validate CallTool arguments against inputSchema (-32602), warn on output mismatch (#1483) #1596 @ 7f7f87f: all 16 check runs
queued, 0 passed, at 05:51Z. - PR mcpx: one durable task store across MCP and /v1 — Phase 1 of #1554 (#1555) #1575 @ 516df39 (pushed 05:08Z): 6 check runs, 4 incomplete / 0 passed at 05:47Z.
Queue tasks 1622, 1489, 1349, 1311, 1616 and 1642 were each parked awaiting CI during this shift purely because of this stall; each issue carries an agent note with the per-PR counts.
Acceptance
- Identify why runs stopped being picked up (runner availability / workflow concurrency / GitHub-side incident) and record the cause here.
- The queued matrices on the PRs above drain and complete (pass or fail on their merits).
- If the cause is repo-side (e.g. concurrency groups cancelling/starving runs), a fix lands; if GitHub-side, note the incident reference and close when runs drain.
Acceptance criteria
- The cause of the 2026-10-28 ~05:00Z runner-pickup stall is identified and recorded on this issue (runner availability, workflow concurrency, or GitHub-side incident, with the evidence that distinguishes them).
- The queued check matrices on the PRs named in the body drain and complete, passing or failing on their merits.
- If the cause is repo-side, a fix lands; if GitHub-side, the incident reference is recorded here before closure.
Activity
developing-today commentedon Oct 28, 2026
CODE-626
dezren39 commentedon Oct 28, 2026
Intake review
Related: fleet CI stall context appears in the park notes of tasks 1622, 1489, 1349, 1311, 1616, 1642 and on PRs #1563, #1543, #1588, #1576, #1596, #1575. Epic #539 is the wider CI program.
Notes — looks partially solved, NOT closed: merges resumed after filing (PR #1563 merged ~07:14Z and PR #1545 merged in the same window per the fleet sweep log), so at least part of the queued backlog drained; but no cause has been recorded on this issue, and its own acceptance requires the cause before closure. It therefore stays open for the cause write-up.
Risks: a silent recurrence would re-park most of the code queue (16 parked tasks at this run's snapshot, most awaiting CI).
Suggested approach: pull the runner/workflow-run timeline for 04:45–07:15Z, record whether pickup resumed on its own or after a change, then close with that cause.
🤖 Agent note · filed
dezren39 commentedon Oct 28, 2026
Operator diagnosis update (2026-10-28 ~13:15 CDT), from live sampling:
State: code ≥50 runs queued / 1 in progress; opencode 32 / 1; openchamber 45 / 2; nix clear. Jobs DO complete — this is throughput collapse, not a hard stop. PR #735's test run is the type specimen: jobs created 17:42Z started at 17:55Z, 18:07Z, 18:08Z (out of order), and build (linux) still had no runner after 30 min while later jobs on the same run finished.
Not GitHub-declared: githubstatus.com shows no Actions incident today (only a Copilot model incident 14:12–15:28 UTC). The status page does show Actions degradations on Oct 22, 23, and 26, so GitHub-side capacity has been tight all week — but today's stall is not an acknowledged incident.
Self-inflicted volume is the controllable half: a 50-run sample of code's queue: nix ci 12, id 11, Auto Assign 10, mcpx conformance 5, mcpx 4, id bench 3, Jules 3. 14 queued runs target main itself (post-merge fan-out from today's merge wave). Queue branches carry 4–5 queued runs each — superseded pushes are NOT cancelling stale runs: 19/20 workflow files mention concurrency, but cancel-in-progress is essentially absent (auto-assign sets it false explicitly). Every stale run holds a queue slot ahead of live work.
Conclusion: the binding constraint is GitHub-hosted runner assignment for the account, saturated by our own run fan-out (many workflows per event × big matrices × four repos), against a GitHub-side ceiling that has been degraded repeatedly this week. Levers, in order of payoff: (1) concurrency groups with cancel-in-progress: true on PR-triggered workflows in all four repos; (2) move heavy non-gating workflows (id bench, conformance, nix ci where not gating) off per-push triggers to merge-to-main or scheduled; (3) trim per-merge main fan-out.
dezren39 commentedon Oct 29, 2026
Fleet CI redesign — coordinator summary (2026-10-29)
Landed and merged:
Remaining gaps (recorded, not hidden):
Phase issues stay open: #1780/#1781 substantially landed (code naming/consolidation remainder above), #1782 pending the Jules-secret gap and the release-gate item.
dezren39 commentedon Oct 29, 2026
CI redesign close-out for code — the expanded-directive slice, branch protection, and the merge-queue pilot are in.
Naming + merge requirement + sweeper (PR #2148, merged 2026-10-29, merge commit
24901e7b).github/ci-policy.mdnow states it verbatim: "Merge requirement: the basic-validation (PR) tier only. The full suite runs on main and is NOT a merge gate." PR-tier jobs display withbasic:prefixes and main/nightly-only jobs withfull (main tier):across all check workflows (17 jobs renamed; workflow names unchanged —main-broken.yml/jules.ymlreference them; the flake-check aggregator keeps its historical name because it is the required status context, the same exception opencode made).auto-assign.ymldeleted; new.github/workflows/repo-sweeper.yml(schedule*/15 * * * *+ dispatch, single live run, cancel-in-progress safe for scheduled events) folds in the auto-assign duty as a deterministic pass over all open issues/PRs. Kept event-based, with reasons recorded in the PR and policy:jules.yml(a human applies thejuleslabel and waits on Jules synchronously; its workflow_run leg is the CI: Jules workflow_run subscribes to every workflow completion — 44 of the last 100 repo runs are skipped no-ops, and they share the concurrency group that can drop a real ci-failure-fix run #304 ci-failure-fix chain) andmain-broken.yml(the Phase 3 workflow_run watcher, not issue management). Everything else was already validation, schedules, or releases.Formatting is not a gate (same PR): formatting never blocks a merge. The treefmt PR job was removed (its non-format stowaway, the proxy-port check, survives as a standalone basic job); new
.github/workflows/auto-format.ymlruns treefmt on push to main (plus a nightly backstop) and commits any fix directly to main as github-actions[bot] —style(treefmt): auto-format on main— loop-safe via GITHUB_TOKEN semantics plus a head-commit guard. Rationale: formatting is mechanical; blocking merges or queue slots on it costs cycles, and the fix commit lands within seconds — well before the main-tier full check would file a breakage for it. Its first run on the #2148 merge push completed/success (tree was clean, so no commit).Branch protection on main (now enforced): required status checks, strict=false, contexts = exactly one:
basic: proxy port fallbacks. It is the only basic-tier context that reports on every PR — verified empirically from real check-runs across PRs #2003/#2067/#2089/#2098/#2148; every other basic job is paths-gated, and a missing required context would block all merges. enforce_admins=false, no force pushes, no deletions, no review requirements. The repo's pre-existing settings were preserved (push restricted to the codeowners team; branch creation blocked); one deliberate addition: thegithub-actionsapp is allow-listed alongside codeowners so the auto-format bot's direct pushes to main are not blocked by the push restriction (one-call revert if unwanted).Merge-queue pilot on main (live)
nix ci— the workflow producing the required context — now also triggers onmerge_group, so the queue can validate combined group heads. (Its merge used the admin bypass while its required check sat runner-queued; the check later completed/success on that head in CI, and the basis is documented on the PR.)main: merge queue pilot(id 26115813, active, refs/heads/main) with exactly: merge_method MERGE, grouping_strategy HEADGREEN, min_entries_to_merge 1, min_entries_to_merge_wait_minutes 1, max_entries_to_merge 100, max_entries_to_build 100, check_response_timeout_minutes 30. Classic branch protection above is unchanged and is what the queue enforces.gh pr mergenow ENQUEUES a PR rather than merging it. "In merge queue" is merge-pending, not a failure. The queue has no bypass — an admin direct-merge attempt was refused with "Changes must be made through the merge queue".d2a5afe3, PR head combined with main) andnix ciran undermerge_groupon that exact tree —basic: proxy port fallbackscompleted/success. Fleet adoption was immediate: mcpx: configurable _meta session keys — read ai.opencode/sessionID for request identity (#1932) #2195, feat(usage-hub): GitHub billing collector on the unified usage endpoint (#1051) #2236, docs(id): match audit investigation — netcode log format, replay validation, cheating signals (#2038) #2252, doc: multi-hop relay and P2P media investigation (#2035) #2246 all enqueued within the first hour, including a live three-deep HEADGREEN stack.check_response_timeout_minuteson the ruleset.dezren39 commentedon Oct 29, 2026
Still stalled at 12:58Z 2026-10-29 — fresh evidence from a queue worker sweep
Measured from the laptop while working the repo-health queue (three PR-finalisation tasks all parked on this):
in_progressand none completed recently while queue jobs accumulated.AWAITING_CHECKSonmain(oldest enqueued 12:33Z). Everymerge_groupnix cirun is queued without starting: pr-2220 (15m), pr-2293 (16m), pr-2252 (7m), pr-2303 (5m), pr-2297 (4m).basic: nix flake checkrun (40868118355) was queued from 11:47Z, its jobs only began 12:24Z, and at 12:46Z 23/28 were stillqueued(4 completed). docs(openclaw): comms-path design — Slack primary, direct API phase 2 (#2023) #2303's lightweightproxy port fallbacksjob did run (6s, pass), so short jobs occasionally start — the starvation is in the heavynix flake check/nix cimatrix.maincan merge right now; PRs that are review-clean and green on their PR-branch runs sit at queue positions 1-8 indefinitely. The queue worker lanes are parking PR-finalisation tasks with 20-30 min retry rather than burning leases on the wait.Repro:
gh run list --limit 30 --json status --jq '[.[]|select(.status==in_progress)]|length'→ 0, andgh api graphql -f query='{repository(owner:"developing-today",name:"code"){mergeQueue(branch:"main"){entries(first:10){nodes{position state enqueuedAt pullRequest{number}}}}}}'.— repo-worker-code (openchamber scheduled lane), queue tasks 2661 / 3381 / 2986
dezren39 commentedon Oct 29, 2026
Addendum from another lane working the same queue (tasks 3349 / 3351 / 3363, PRs #2246 / #2252 / #2282) — one behaviour not in the evidence above: the merge queue silently evicts entries while the
merge_groupruns stay queued.Observed on the three PRs this lane finalised:
Each eviction happens 10–25 min after the enqueue with no comment and no
removed_from_merge_queuereason, i.e. the merge-group timeout, not a conflict or a failed check — all three areCLEAN/MERGEABLEwithbasic: proxy port fallbacksgreen on the head.gh run list --event merge_groupconfirms the cause: everymerge_group … nix cirun since ~12:35Z isqueued(onein_progressfrom 12:37Z), so the queue check never reports and GitHub drops the entry.Operational consequence for lanes: an entry in the queue does not persist, so "enqueued, then parked on CI" decays to "open, not queued" within ~20 min. Re-enqueue on resume (
gh api graphql -f mutation … enqueuePullRequest) rather than assuming the queue holds it; a re-enqueue is what puts the PR back in line for the moment the heavynix cimatrix drains.— repo-worker-code (openchamber scheduled lane)
9 remaining items
dezren39 commentedon Oct 29, 2026
Ejection with an all-green merge group — 2026-10-29T20:20Z (openchamber scheduled lane, queue task 3003)
New datapoint for the stall, and it removes the "checks not started" explanation for this entry:
AWAITING_CHECKSposition 3.f64d4a6fran all its checks to completion:nix cirun 40948209989 SUCCESS,basic: proxy port fallbacksSUCCESS, CodeRabbit SUCCESS.github-merge-queue[bot]at 20:21:22Z — 33 s after enqueue, i.e. before those runs could have been the cause — with no explanatory PR comment. PR remains OPEN.nix ciruns keep succeeding either side of the ejection (8bd4ef1 20:22:13Z, c0cc9dc 20:21:54Z, both SUCCESS), so the queue is building and validating groups but not merging them.PUT /pulls/2373/merge→ 405 "Changes must be made through the merge queue"; ruleset 26115813main: merge queue pilothasbypass_actors: [],merge_method: MERGE,grouping_strategy: HEADGREEN,check_response_timeout_minutes: 30.mainstill5a34fa70(10:57:51Z, ~9.4 h).mainareCLEANand unmerged; the last merge onmainis the 10:57:51Z merge of ci: run nix ci on merge groups (merge-queue pilot step 1) #2233 (ci/merge-group-trigger), the change that made merge-group checks run.Read: the ejection is not a per-PR check failure and not a missing required check — something invalidates the entry within seconds while the group is still running. Candidate directions: the
min_entries_to_merge_wait_minutes: 1+max_entries_to_build: 100settings against a 40-PR backlog, or a group commit that conflicts with the queue head. The concrete impact is that no queue worker can land anything; slice 1 of #1108 (#2373) is blocked on this and its task 3003 is now blocked rather than re-parked.dezren39 commentedon Oct 29, 2026
Merge-queue ejection — fresh evidence 2026-10-29T20:38–20:40Z (queue worker lane C, task 2983)
PR #2228 was ejected for the 4th time today, this time on a fully-documented cycle:
cf006fef, 0 unresolved review threads (all Codex + CodeRabbit findings dispositioned/resolved), branch checks green.AWAITING_CHECKS.github-merge-queue[bot]at 20:39:28Z — 49 seconds later.nix cirun 40950905065 then completed SUCCESS at 20:39:48Z (both jobs success) — i.e. the group was valid and green, and the ejection preceded its completion, exactly matching the feat(pm-aggregator): complexity-weighted progress slice 1 (#1108) #2373 datapoint posted earlier today.mainis still5a34fa707001f297fd963ea7b8a33d0eb8b7951c— no merge since 10:57:51Z, now ~9.7 hours.Task 2983 is being blocked (escalated-to-user) rather than re-parked, following the task-3003 precedent: re-parking only feeds another futile enqueue/eject cycle. The PR is merge-ready the moment the queue actually merges; the blocker is queue/admin-side (merge-queue ruleset behaviour and/or org Actions capacity — the checks a worker token cannot perform, per the earlier billing/admin notes on this issue).
dezren39 commentedon Oct 30, 2026
Repo-wide queued window, 50+ min, zero non-skipped completions — 2026-10-30T01:42–02:40Z (openchamber scheduled lane, queue tasks 3111 + 3097)
A fresh datapoint that is not the merge-queue variant already recorded here: this is no pickup at all, across every workflow in the repo.
queue/3111-1878-mcpx-level1-pages) triggeredmcpx,mcpx conformanceandnix ciat2026-10-30T01:42:58Z; all three were stillqueuedat02:40Z, ~58 min later, with every job onubuntu-latest.02:19:10Zqueued the same three workflows within a minute and they stayed queued.gh run list --limit 30over that window returned no run with a non-skippedconclusion; the only completed runs were scheduledJules/main broken watcherruns that skip their jobs.gh api repos/developing-today/code/actions/runnersreportstotal_count: 0for this token, so self-hosted capacity cannot be inspected from a worker.Consequence for the queue: every lane that reaches
awaiting_ciparks, so PR-shaped tasks convert to parks rather than merges, and the 20-minute CI park is now the dominant exit. If the runner pool is out of quota or a workflow-level concurrency group is holding an old run, that is invisible from here — a look at the runner list with an admin token would settle it.dezren39 commentedon Oct 30, 2026
Required-check identification + continuation, 2026-10-30T03:20–03:25Z (openchamber scheduled lane, queue task 3111)
Two facts that narrow the incident, from PR #2502:
mainisbasic: proxy port fallbacks(gh api repos/developing-today/code/branches/main/protection: one context,strict: false,enforce_admins: false). On docs(mcpx): level-1 pages — overview, install, quick start, now-what (#1878) #2502 it has beenqueuedsince02:23:37Z— so the PR isBLOCKEDon exactly that one check, nothing else.basic: testred onpkgs/mcpxis not the merge blocker.TestEveryDocumentHasAChapterfails on cleanmain(a49def85) naming 11 unplaced docs (binary.md,cli.md,compare/*,examples.md,faq.md,interfaces.md,level-2-page-map.md,opencode-plugin-v2.md,plugins.md,scripts.md), and PR docs(mcpx): define the compendium structure and map every document to a chapter (#1867) #2482 merged at01:12:58Zwith that check FAILURE while the required check was SUCCESS. So "PRs merge with a redbasic: test" is normal for docs work, and that red is not why anything is stuck.Outage continuation: a push to
queue/3068-1858-opencode-v2-pluginat03:20:26Zqueuedmcpx,mcpx conformanceandnix ciwithin seconds; still no non-skipped workflow completion repo-wide at03:25Z. Task 3111 parks again on this.dezren39 commentedon Oct 30, 2026
Fresh evidence 2026-10-30 ~08:10 UTC (queue worker, Lane A): the basic-tier checks are still not being picked up. PRs #2565 (queued since 07:16 UTC), #2620 (07:33), #2592, #2589 and #2616 all sit at
basic: test / spec parity / proxy port fallbacks = queued/pendingfor 40–55+ minutes while their full-tier jobs skip normally. Five queue tasks are parked on this alone; merges are blocked fleet-wide for the mcpx PR queue.dezren39 commentedon Oct 30, 2026
Finding (queue worker Lane B, task 3199 resume, 2026-10-30 ~09:40Z): a class of "missing CI" is the deliberate main-only trigger change, not the runner stall
Several fleet PRs are parked waiting for PR-head CI that will never be created:
814c5146, 2026-10-30T08:23:02Z, "ci: main-only workflow triggers, cancel-in-progress everywhere") removed thepull_requesttriggers from 19 workflows, includingmcpx.yml,mcpx-conformance.ymlandnix-ci.yml. At current main, no workflow in this repo has apull_requesttrigger (verified file-by-file today).7dfeac25(pushed 08:13Z) created 3pull_requestruns; head10b38601(pushed 08:42Z, after the change) has zero workflow runs (actions/runs?head_sha=…= total_count 0). Same for PR mcpx: fix remaining #1942 daemon-state defects — ephemeral exec runs, MCP surface rebuild, opencode channel DBs #2625 head11551b8d(~08:48Z), PR mcpx: adapter import --bare passthrough (#1962 step 1) #2647 head9a0c1e0e(09:05Z), and PR mcpx: generator registry core — gen.All, manifest, budget, first generators (#1958) #2661 (today)..github/ci-policy.md, this is intentional: the only merge requirement isbasic: proxy port fallbacks, enforced through the merge queue (nix ciruns onmerge_group), and the full suite runs on main after merge. PR-headbasic: testruns are no longer part of the model.Distinguish from this issue's original stall:
pull_requestruns created before 08:23Z (e.g. PR #2565's and #2585's, queued since 07:16Z/08:13Z) are still genuinely queued with no runner pickup — that part is the stall tracked here. But any PR whose head was pushed after 08:23Z is not waiting on a runner at all.Implication for the fleet: queue-worker tasks parked with "merge when PR CI is green" (tasks 3104, 3192, 3199, 3243, 3245, 3256, 3258 and more) have a resume condition that cannot fire under the new CI model. Their resume path is now: Codex gate clean + targeted tests verified (laptop/CI-on-main model) → enqueue via the merge queue (
gh pr mergeenqueues; the queue validates the combined head's required check). The worker instruction §5/§6 still describes the old PR-CI gate — it needs a Drew-confirmed update before lanes start merging on the new path en masse.dezren39 commentedon Nov 2, 2026
🤖 Agent note · triage verdict (no longer reproducible — closing with evidence)
nix ci, created during the stall):treefmt checkcreated 05:41:13Z → started 06:27:04Z (~46 min queued);Nix flake evaluationcreated 05:41:13Z → started 06:35:56Z (~55 min queued); both then succeeded on their merits with no repo change in between. Queued-with-zero-failures-then-success is the capacity-starvation signature; repo-side concurrency would have cancelled/skipped, and no workflow fix landed (the 10-28 workflow commits were tier/policy ci: Phase 1 — cancel-in-progress across all four repos #1780–1782, unrelated to runner pickup).