Skip to content

[aw-failures] [aw-fix] P0: Find and stop the workflow_dispatch storm hitting Approach Validator + Daily Syntax Error Quality Check #48813

Description

@github-actions

Stop whatever is bulk-dispatching these two workflows — that's what blew both daily AI-credit caps and cascaded into 57 straight failures.

Fixing the guardrail message alone won't help; the dispatcher is the actual bug.

Problem statement

In a single ~1.5h window (2026-07-29 ~04:35–06:05 UTC), Daily Syntax Error Quality Check (.github/workflows/daily-syntax-error-quality.lock.yml) and Approach Validator (.github/workflows/approach-validator.lock.yml) were each invoked roughly 29 times via workflow_dispatch. Neither workflow's triggers explain that volume:

  • daily-syntax-error-quality.md: schedule: daily + workflow_dispatch — should fire once/day from cron, not ~29x.
  • approach-validator.md: label_command (approach-proposal, needs-design labels) + slash_command — normally event-driven per issue/PR, not a tight burst.

Once each workflow's daily AI-credit budget was exhausted by the burst (GH_AW_MAX_DAILY_AI_CREDITS: 5000 for daily-syntax-error-quality, 10000 for approach-validator), every subsequent run that day failed at the Check daily workflow token guardrail step before ever calling the model — 57 of 60 unique failed runs in the 6h window.

Affected workflows and run IDs

  • Daily Syntax Error Quality Check — representative failed run §30427016013
  • Approach Validator — representative failed run §30427016566
  • Comparator/baseline (last known success): run 30341316460 (2026-07-28, Daily Syntax Error Quality Check)
  • Full list of 57 affected run IDs is in the parent issue's evidence section.

Probable root cause

One of:

  1. A bot/automation is re-dispatching these workflows repeatedly without dedup (e.g., re-applying approach-proposal/needs-design labels to many issues/PRs in a batch, or a retry loop with no backoff on failure).
  2. A manual or scripted bulk workflow_dispatch API call loop (e.g., a test or backfill script) targeting these two workflows specifically.

audit-diff on the representative run confirms the failure is a clean pre-flight abort, not a network/firewall issue: total_requests 1→0, token_usage_change -100%, firewall_diff shows 0 anomalies — so the model/API layer was never reached; only the guardrail check ran and failed.

Proposed remediation

  1. Pull triggering_actor and event payload for the 57 affected run IDs (or a representative sample) via the GitHub Actions API to identify the actual dispatcher.
  2. If it's a bot/automation: add dedup/throttling so the same workflow isn't re-dispatched for the same target within a short window, and add exponential backoff on retry.
  3. If it's a legitimate bulk operation: raise the daily AI-credit cap for that operation ahead of time, or stagger the dispatches so the guardrail doesn't cascade-fail the whole batch.
  4. Separately, consider making the guardrail failure mode softer — e.g., skip/queue instead of hard-fail with exit 1, so a capacity-exhaustion event doesn't read identically to a real workflow bug in monitoring.

Success criteria / verification

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 115.9 AIC · ⌖ 21.8 AIC · ⊞ 5.1K ·

  • expires on Aug 4, 2026, 11:53 PM UTC-08:00

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions