Skip to content

CI: pushes to PR #2629 after opening created no workflow runs (synchronize events dropped) #2713

Closed

Description

@dezren39

Problem

Pushes to an open PR stopped creating GitHub Actions workflow runs, while other PRs' synchronize events in the same window created runs normally.

Evidence (2026-10-30, developing-today/code PR #2629)

  • PR opened 08:15Z at 7530992a → runs created normally (nix ci success, mcpx conformance success, mcpx test job ran).
  • Push 49fe420c at 08:46Z → no workflow runs created (only a CodeRabbit check run on the head).
  • Push 684a1339 at 09:47Z → no workflow runs created.
  • Push 43f24bd7 at ~11:26Z (empty commit from a laptop HTTPS push, message has no skip-ci marker) → no workflow runs created; commits/43f24bd7/check-runs total = 1 (CodeRabbit only).
  • Control: at 11:23Z, PR branch queue/3270-1973-flake-policy received a pull_request mcpx run normally, and main pushes created runs normally — Actions itself was up.

Impact

PR-tier CI silently absent on later commits of an affected PR: the head looks "pending forever" (combined status pending, zero status contexts), and a worker cannot tell "queued" from "never triggered" without querying check-runs per head SHA.

Acceptance

  • Root cause identified (trigger config, concurrency-group suppression, token/event quirk, or GitHub-side drop) and recorded here.
  • Affected PRs either re-trigger automatically or there is a documented, reliable retrigger path that does not require an empty commit (the empty commit did not work here).

Activity

  1. added
    kind/ciCI/CD pipelines, workflows, and build automation
    priority/highImportant and time-sensitive
    effort/mMedium: expected under 1 day of work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 30, 2026
  2. dezren39 commented on Oct 30, 2026

    @dezren39
    MemberAuthor

    Intake review

    Fleet monitor intake pass (2026-10-30 run). Labels normalised this pass: kind/ci, priority/high, effort/m, status/needs-triage, meta/enriched. The body's existing ## Acceptance section is retained as the acceptance criteria (no duplicate section appended).

    Still live: the evidence window is today (PR #2629 pushes at 08:46Z, 09:47Z and ~11:26Z created no workflow runs while a control PR branch and main pushes triggered normally at 11:23Z), and no merged change since filing addresses dropped synchronize events. Priority set to high: a PR whose head silently has no CI looks "pending forever", and workers cannot distinguish queued from never-triggered without per-SHA check-run queries — this undermines the fleet's CI-first verification rule.

    Notes / risks: the control observations rule out a fleet-wide Actions outage; the discriminating suspects are per-PR/per-branch state (e.g. a workflow-file or concurrency-group interaction on that branch) or a GitHub-side event drop for that PR. The empty-commit retrigger failing is the strongest clue — it rules out the usual "just push again" remedy.

    Suggested approach: start from the GitHub API audit trail for PR #2629 (event timeline + the head SHAs' check-runs), compare against the control branch's pull_request run at 11:23Z, and test the retrigger paths that do not need an empty commit (close/reopen, workflow_dispatch where defined) so the runbook answer is empirical.

  3. added
    queue/queuedHas a queued task in the Supabase agent queue
    source/creatorFiled by the new-issue-creator job
    task/triageQueue packet kind: verify, reproduce, and route this item
    on Nov 3, 2026
  4. developing-today commented on Nov 3, 2026

    @developing-today
    Contributor

    agent-note:v1
    state: reviewed (coverage loop-back)
    Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7349 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
    Found via: coverage loop-back 2026-11-03

  5. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026
  6. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  7. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done (triage — closing with the root cause recorded)

    • Agent: queue worker (repo-worker-code), Lane P shift (lane code-p)
    • Session / ID: queue task 7349 (repo-health-code-pilot)

    Root cause — the PR was in a merge-conflict state for the sustained outage

    Reconstructed after the fact with git merge-tree (laptop clone) of each PR head against the contemporary main, cross-checked against the surviving run list for branch queue/3236-1939-mcpx-mdbook:

    Push (2026-10-30) Head vs contemporary main Workflow runs
    PR opened 08:15Z, head 7530992a clean vs main@08:15 (7dfeac25) created (nix ci / mcpx / mcpx conformance, 08:15:43Z) ✔
    08:46Z, head 49fe420c clean vs main@08:46 (10b38601, 11551b8d) none created — see the caveat below
    09:47Z, head 684a1339 CONFLICTING vs main@09:47 (cc529762) none created ✔ explained
    ~11:26Z, head 43f24bd7 (empty commit) CONFLICTING vs main@11:26 (b3e9fb9c) none created ✔ explained
    ~11:30Z, merge of origin/main into the branch mergeability restored mcpx pull_request runs created 11:30:14Z and 11:32:45Z ✔

    The conflict is a single file: pkgs/mcpx/docs/compendium.manifest.json — the branch kept registering docs in the manifest while main-side mcpx work edited the same file. Even the unchanged head 49fe420c is conflicting against main@09:47: main moved into conflict with the branch during that hour, no branch push required.

    Mechanism: pull_request workflow runs execute against the PR's merge commit (refs/pull/<n>/merge). When the PR has merge conflicts, GitHub cannot compute that commit and creates no runs at all — no failed run, no pending status, just the CodeRabbit app check (an app check run, not an Actions workflow). That is exactly the "pending forever, zero status contexts" signature in the original evidence, and it is why the control PR at 11:23Z (not conflicted) triggered normally while Actions was otherwise up. Trigger config is ruled out: at the then-current main (9defedd3), nix-ci.yml had an unfiltered pull_request: trigger, and mcpx.yml's paths: filter cannot explain a missing nix-ci run.

    Caveat, recorded honestly: the 08:46Z drop is not explained by the conflict — the head was mergeable at every main state that can be reconstructed for that minute. It stands as a single-event GitHub-side drop (this was also the era of multi-hour runner backlogs: the 08:15 runs took ~3h to complete). It is not retroactively provable either way, and no trigger-config or concurrency explanation fits it (the nix-ci concurrency group cancels/replaces, it never silently swallows a run creation). The sustained outage — every push from some point in the 08:46–09:47 window until 11:30Z — is the conflict.

    The reliable retrigger path (acceptance item 2)

    • Merge current main into the PR branch (or rebase) and push. Restoring mergeability is what makes synchronize fire again — demonstrated live here at 11:30Z. This is the documented path.
    • An empty commit does NOT retrigger while the PR is conflicted (proven by 43f24bd7): there is still no merge commit to run against. "Re-run" buttons and empty pushes are equally useless in this state.
    • Diagnostic shortcut for the next occurrence: before hunting Actions, check the PR's mergeable state and commits/<head>/check-runs — conflicted + only non-Actions app checks = this issue, not an Actions outage.

    Why this closes now

    The incident class is also retired by policy: on 2026-11-02, commit 285e425b ("ci: no PR triggers — workflows run on main push, schedule, or manual dispatch only") removed PR-tier triggers fleet-wide, and main-push CI plus the main-broken sweep are the standing detection path. PR #2629 itself merged 2026-10-30T11:33Z. Queue task 7349 completes as triaged with this record; closing the issue on its merits — root cause recorded, retrigger path documented, one single-event drop explicitly left unexplained rather than papered over.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort/mMedium: expected under 1 day of workkind/ciCI/CD pipelines, workflows, and build automationmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivesource/creatorFiled by the new-issue-creator jobtask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions