[aw-failures] [aw] Failure Investigator Report — 2026-07-29 (6h)

[github-actions]

Executive Summary

Fix check_rate_limit.cjs's pagination loop first — it doesn't stop once it passes the rate-limit time window, so it walks the AI Moderator workflow's entire run history every single time (confirmed: page 250, runs from January 2026) until CI kills the step after ~15 minutes. This alone caused 7 of 19 failures (37%) this window.

Fix Avenger's dependency build second — actionlint@v1.7.12 fails to compile against the resolved yaml module, 100% reproducible, 3/3 runs.

AWI_MAINTENANCE_TOKEN is still broken — #48899 is not fixed. Two more add_comment 401s this window, plus a newly-found update_pull_request 403 on the same PR. Updated that issue directly with the new evidence rather than duplicating here.

GitHub Copilot rejected model access twice ("No model available. Check policy enablement"), killing PR Code Quality Reviewer mid-run when it tried to launch a background review sub-agent. This is an external Copilot-service policy issue, not a gh-aw code bug — documented below, not ticketed.

• 19 failed/cancelled runs in the last 6h across 8 workflows (full detail below; prior report's data gap is closed — all 19 runs were resolved via direct API lookups, not just the 5 in the pre-fetch sample).
• 2 new P0 clusters root-caused and sub-issued.
• 1 P1 already tracked in [aw-failures] [aw] PR Sous Chef: add_comment failing with 401 Bad credentials (AWI_MAINTENANCE_TOKEN) #48899 — updated with new evidence, kept open.
• 1 P0-by-volume cluster is external (Copilot backend) — documented, not ticketed.
• 4 single-occurrence failures across 4 different workflows — no repeating signature, monitor only.

Failure Cluster Table

Cluster	Severity	Workflow	Occurrences	Representative Run	Comparator Run	Status
ai-moderator-ratelimit-pagination	P0	AI Moderator	7/19	§30473844593	§30460703500	New sub-issue filed
avenger-actionlint-yaml-build	P0	Avenger	3/19	§30468512211	§30464375707	New sub-issue filed
pr-sous-chef-token	P1 (tracked)	PR Sous Chef	3/19	§30456200459	§30465347575	Updated #48899 — still unresolved
pr-code-quality-copilot-model	P0 (external)	PR Code Quality Reviewer	2/19	§30479767476	§30470606972	Documented below — not ticketed (external Copilot backend policy)
isolated-singletons	P2	Delight, Daily Agent of the Day Blog Writer, Super Linter Report, CI Optimization Coach	4/19	§30465450611	—	Monitor — no repeating signature

Evidence

AI Moderator — unbounded pagination (root cause + audit-diff)

Job log (pre_activation, "Check user rate limit", run 30464716556) shows the check paging back through 250 pages of workflow-run history to January 2026 before being force-cancelled:

Skipping run 20969558885 - created before threshold (2026-01-13T19:21:13Z)
Fetching page 250 (up to 100 runs per page)...
##[error]The operation was canceled.

Root cause in actions/setup/js/check_rate_limit.cjs (lines ~159-164):

if (runCreatedAt < thresholdTime) {
  core.info(`   Skipping run ${run.id} - created before threshold (${run.created_at})`);
  continue;   // keeps scanning instead of stopping — bug
}

listWorkflowRuns returns runs newest-first, so once one run is older than the threshold, every run after it (this page and all following pages) is also older. The loop should stop there; instead it keeps paginating for the full run history every single invocation.

audit-diff between two cancelled runs ([§30473844593] vs [§30460703500]) shows zero firewall/network drift — same deterministic behavior both times, ruling out environmental flakiness.

Avenger — actionlint/yaml build failure

Identical compile error across all 3 runs:

../../../go/pkg/mod/github.com/rhysd/actionlint@v1.7.12/action_metadata.go:273:22: te.Errors[0].Error undefined (type string has no field or method Error)
../../../go/pkg/mod/github.com/rhysd/actionlint@v1.7.12/parse.go:1555:59: e.Err undefined (type string has no field or method Err)
../../../go/pkg/mod/github.com/rhysd/actionlint@v1.7.12/parse.go:1567:26: undefined: yaml.ParserError
Process completed with exit code 2.

go.mod carries three yaml-family modules (gopkg.in/yaml.v3 v3.0.1, go.yaml.in/yaml/v3 v3.0.4, go.yaml.in/yaml/v4 v4.0.0-rc.6) — actionlint@v1.7.12 is resolving to one that doesn't provide the yaml.ParserError type/structured error fields it expects.

PR Code Quality Reviewer — Copilot "No model available" (external, not ticketed)

[copilot-sdk-driver] [sdk-driver] error: Execution failed: Error: No model available. Check policy enablement under GitHub Settings > Copilot
[copilot-harness] all 3 retries exhausted — giving up (exitCode=1)

Both failures ([§30479767476], [§30470606972]) happened after the main review completed successfully, when the agent tried to launch a background sub-agent (task tool) for a follow-up review pass — that nested Copilot session couldn't get a model allocated. This is a GitHub Copilot backend/policy condition, not a gh-aw defect. Two occurrences don't yet establish a stable pattern; if it recurs next window, file a sub-issue to harden the workflow against background-task failures (e.g. don't fail the whole run when only the optional follow-up sub-agent errors).

Existing Issue Correlation

• #48899 — still active, not fixed. Updated with 2 new occurrences and the new update_pull_request 403 symptom.
• #48897 (this issue) — refreshed as parent for this window's 2 new sub-issues.
• #48821 — unrelated (Daily Evals Feature Report), left untouched.

Fix Roadmap

P0:

1. check_rate_limit.cjs — break the pagination loop (hasMore = false + exit) the moment a run older than the threshold is seen. See sub-issue.
2. Avenger's yaml/actionlint module graph — align actionlint's version with the resolved yaml module, or vice versa. See sub-issue.

P1:
3. Confirm the AWI_MAINTENANCE_TOKEN replacement (per #48899) also carries the workflows OAuth scope — the new 403 on PR #48593 needs it, or it'll fail again right after rotation.

P2 (monitor):
4. GitHub Copilot "No model available" — external. Re-evaluate if it recurs.
5. Delight, Daily Agent of the Day Blog Writer, Super Linter Report, CI Optimization Coach — one occurrence each this window, no shared signature. No action.

Sub-Issues Created

• AI Moderator rate-limit pagination timeout (P0)
• Avenger actionlint/yaml build failure (P0)

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 180.4 AIC · ⌖ 24.4 AIC · ⊞ 5.5K · ◷

---

Update — 2026-07-30 01:14 UTC (6h window)

Fix PR Code Quality Reviewer's background sub-agent pattern — it just took down 2 more runs, exactly as predicted last window. New sub-issue filed. Fix the conclusion job's checkout retry — a closed April bug (#28149) just regressed. New sub-issue filed. AWI_MAINTENANCE_TOKEN is fixed — closed #48899 with evidence.

• 4 failed runs across 3 workflows this window (failed_run_ids: 30496744608, 30496052606, 30486275442, 30483968883).
• 1 issue closed as fixed (credential rotation confirmed working).
• 2 new P1 sub-issues filed on this parent.
• 1 failure already self-tracked by the workflow's own auto-filer — no action needed.

Failure Cluster Table

Cluster	Severity	Workflow	Occurrences	Representative Run	Status
pr-code-quality-bg-subagent	P1	PR Code Quality Reviewer	2/4	§30496052606	New sub-issue filed
conclusion-checkout-dns-cascade	P1	PR Sous Chef	1/4	§30483968883	New sub-issue filed (regression of closed #28149)
pr-sous-chef-token	—	PR Sous Chef	0 new this window	—	Closed #48899 — confirmed fixed
byok-ollama-pricing	P2	Daily BYOK Ollama Test	1/4	§30496744608	Already tracked: #49007 (auto-filed, config issue — qwen2.5:0.5b has no AI-credits pricing)

Evidence

PR Code Quality Reviewer — two failure modes of the same background sub-agent pattern

Run 30496052606: the task-tool sub-agent grumpy-coder never finished — polled at 253s/431s/628s/751s elapsed, still status: running every time. The main session then hit its own idle watchdog: [copilot-sdk-driver] error: Timeout after 870000ms waiting for session.idle, failureClass=sdk_session_idle_timeout, job killed after 14.5 min with zero output.

Run 30486275442: the same background-sub-agent pattern failed differently — all 4 harness retry attempts hit Error: No model available. Check policy enablement under GitHub Settings > Copilot immediately, failureClass=partial_execution x4, retriesRemaining=0. This is the exact signature flagged (but not ticketed) in this issue's prior update — now confirmed recurring.

PR Sous Chef conclusion job — DNS cascade regression of #28149

##[error]fatal: unable to access 'https://github.com/github/gh-aw/': Could not resolve host: github.com
...
Error: Cannot find module '/home/runner/work/_temp/gh-aw/actions/setup_globals.cjs'
##[error]Unhandled error: Error: Cannot find module '/home/runner/work/_temp/gh-aw/actions/setup_globals.cjs'

Identical two-stage cascade to #28149 (closed 2026-04-23): transient DNS failure during the conclusion job's helper-script checkout cascades into an unhandled MODULE_NOT_FOUND. The agent and safe_outputs jobs of this same run succeeded fully — only the conclusion/reporting path broke.

AWI_MAINTENANCE_TOKEN — confirmed fixed, issue closed

Run 30483968883's safe_outputs job: add_comment 2/2 succeeded, update_pull_request 2/2 succeeded, zero 401/403 errors. Closed #48899 with this evidence.

Existing Issue Correlation

• #48899 — closed this window, fixed.
• #49007 — already covers the BYOK Ollama failure 1:1 (auto-filed same run), no duplicate needed.
• #28149 — closed in April, same root cause as the new conclusion-job sub-issue; referenced there as a regression.

Fix Roadmap

• P1: Bound/de-fatal the PR Code Quality Reviewer background sub-agent (timeout + graceful degradation) — see new sub-issue.
• P1: Retry (or bundle) the conclusion job's helper-script checkout — see new sub-issue; reuses [aw-failures] Conclusion job brittle when GitHub returns HTTP 500/DNS errors — cascades to setup_globals.cjs not found #28149's already-scoped remediation.
• P2 (monitor only): BYOK Ollama pricing config — already tracked in [aw] Daily BYOK Ollama Test has no AI credits pricing for model (qwen2.5:0.5b) #49007, no new action.

Sub-Issues Created This Window

• Harden PR Code Quality Reviewer against background task-tool sub-agent failures
• Conclusion job's setup_globals.cjs MODULE_NOT_FOUND cascade has regressed (repeat of [aw-failures] Conclusion job brittle when GitHub returns HTTP 500/DNS errors — cascades to setup_globals.cjs not found #28149)

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 161.9 AIC · ⌖ 20.7 AIC · ⊞ 5.2K · ◷