Skip to content

auto-format: scheduled runs not firing since the 15-minute redesign; record_success can never work with GITHUB_TOKEN (no variables permission scope) — laptop sync installed as the mechanism #4787

New issue
New issue
Closed
Closed
auto-format: scheduled runs not firing since the 15-minute redesign; record_success can never work with GITHUB_TOKEN (no variables permission scope) — laptop sync installed as the mechanism#4787
Assignees
dezren39
Labels
area/ciCI/CD pipelines, workflows, and build automation (area)effort/mMedium: expected under 1 day of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivesource/ci-watchFiled by the main-ci-failure-fix-trigger (fleet CI watch)task/small-fixQueue packet kind: bounded fix with reproduction and targeted tests

Description

@dezren39
dezren39
opened on Nov 3, 2026

Two findings from the 2026-11-03 investigation of the auto-format redesign (#4731)

1. Scheduled runs have not fired since the redesign merged

  • PR ci(auto-format): scheduled passes + last-success variable #4731 merged 2026-11-03T02:17:54Z, switching auto-format.yml to schedule: "*/15 * * * *" + workflow_dispatch.
  • Since then: zero schedule runs (checked repeatedly through 2026-11-03T07:47Z, ~22 slots). The last schedule run is 2026-11-02T16:33:52Z, under the old design.
  • The workflow's state is active, and other workflows' schedules in this repo fire normally (e.g. repo sweeper, 2026-11-03T03:56:36Z). workflow_dispatch runs work fine.
  • Under the old design the nightly run also fired 7–10h late (cron 47 6 * * * landing 13:05–16:33Z), so GitHub's scheduler appears to heavily delay this workflow specifically. Unknown whether the */15 schedule will fire late/bunched or never.
  • Impact: formatting passes currently happen only when someone dispatches one. Drift accumulates in the meantime (formatting never gates merges, by design).

Options: (a) accept and drive passes from the laptop sync timer below (it can also dispatch), (b) investigate repo/org Actions settings for schedule throttling, (c) wait and see whether GitHub's scheduler starts firing late.

2. record_success can never succeed with the workflow's GITHUB_TOKEN

  • The workflow's record_success PATCHes AUTO_FORMAT_LAST_SUCCESS_SHA with the GITHUB_TOKEN. It failed in the verification run (41565232921) and will always fail: GitHub's workflow permissions: vocabulary has no key for repository variables.
  • Proven empirically 2026-11-03 on throwaway branches: adding variables: write or vars: write to permissions: makes GitHub reject the workflow file at parse time — failed to parse workflow: Unexpected value 'variables' (HTTP 422 on dispatch). Probe branches deleted after the test.
  • The REST variables endpoints require a user-grade credential: a fine-grained PAT or GitHub App with the Variables repository permission, or a classic PAT with repo scope. The workflow token cannot be granted it.

Mitigation installed (2026-11-03)

Correction to the #4731 verification record

Run 41565232921 did not push a bot commit: treefmt traversed the 4 candidate files and changed 0 (formatted 0 files (0 changed) / "treefmt made no changes"), then record_success hit the warning path described above. Earlier reports saying it "formatted 4 files and pushed" were wrong.

Related: #2014 (fleet-wide format auto-fix tracker). The workflow-only-exit gap in record_success was already fixed by #4752.

Acceptance criteria

  • A decision is recorded on this issue for finding 1: either the */15 schedule is observed firing (late/bunched counts, with run ids), or passes are driven by the laptop sync timer dispatching the workflow, or an Actions-settings cause is identified and fixed.
  • record_success's mechanism is settled: either the workflow no longer attempts the variables PATCH with GITHUB_TOKEN (the laptop sync timer is documented in the workflow/repo as the mechanism of record), or a credential with the Variables permission is provisioned and a verification run shows the variable advancing.
  • The AUTO_FORMAT_LAST_SUCCESS_SHA variable only ever holds a verified commit SHA in this repo, advanced monotonically by the sync timer (or its successor).
  • The correction to the ci(auto-format): scheduled passes + last-success variable #4731 verification record stands: no report claims a bot commit or formatted-file count that the run log does not show.

Activity

  1. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    Third finding (2026-11-03): biome fails on the mcpx / pm-stack files in range

    The laptop sync from the issue body is installed and verified (first run 2026-11-03T07:46:36Z repaired the variable a56101c3… → 8bd11c59). A loop test then dispatched a fresh pass — run 41598610543, head 9eee21e6 — which consumed the repaired variable correctly (diff path from 8bd11c59) but failed:

    ERRO formatter | biome: failed to apply with options '[format --write
  pkgs/mcpx/internal/defaults/defaults.json
  pkgs/mcpx/plugin/opencode/mcpx/ops.gen.ts
  pm-stack/aggregator/src/actions.ts
  pm-stack/aggregator/src/config.ts
  pm-stack/aggregator/src/derive.ts
  pm-stack/aggregator/src/events.ts
  pm-stack/aggregator/src/pages.ts
  pm-stack/aggregator/src/program.ts
  pm-stack/aggregator/src/projects.ts
  pm-stack/aggregator/src/server.ts
  pm-stack/aggregator/src/sources/live.ts
  pm-stack/aggregator/src/work-items.ts
  pm-stack/aggregator/test/actions.test.ts
  pm-stack/aggregator/test/board-in-flight.test.ts
  pm-stack/aggregator/test/events.test.ts
  pm-stack/aggregator/test/orchestration.test.ts
  pm-stack/aggregator/test/work-items-1206.test.ts]': exit status 1
Error: failed to finalise formatting: formatting failures detected

    So biome exits 1 on at least one file in that batch (the batch fails together; the culprit file needs a bisect — generated ops.gen.ts and defaults.json are the obvious suspects). Until a pass succeeds end-to-end, the variable cannot advance past 8bd11c59 and every pass re-checks the growing range — the sync holds correctly on failure, by design.

    This also means the earlier verification run's clean result was partly luck of range: its 4 candidate files were outside biome's set (and treefmt changed 0 of them).

  2. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    main-ci watch — new main failure for this workflow

    Reusing open issue #4787, which already tracks this workflow's main-branch failure: its 'Third finding (2026-11-03)' comment documents this exact run (41598610543, head 9eee21e) and its failure — the biome formatter fails to apply on the mcpx / pm-stack files in range from 8bd11c5 ('Code formatting aborted due to parsing errors'), so the treefmt job fails with 'failed to finalise formatting: formatting failures detected'. This fix track targets that biome/treefmt failure (finding 3); findings 1-2 of #4787 (schedule not firing, record_success variable) are separately mitigated via the laptop sync timer named on the issue. No duplicate issue filed, per the trigger dedupe rule.

    Filed by main-ci-failure-fix-trigger (fleet). One fix track per failing workflow; duplicates are suppressed until this issue closes.

    🤖 Agent note · filed

  3. dezren39
    added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    queue/queuedHas a queued task in the Supabase agent queue
    task/small-fixQueue packet kind: bounded fix with reproduction and targeted tests
    status/needs-triageNot yet triaged; needs a first look
    priority/highImportant and time-sensitive
    source/ci-watchFiled by the main-ci-failure-fix-trigger (fleet CI watch)
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    status/in-progressQueue worker lifecycle state
    and removed
    status/needs-triageNot yet triaged; needs a first look
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026

  4. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    🤖 Agent note · picked up

  5. dezren39
    added a commit that references this issue on Nov 3, 2026

    Merge pull request #4794 from developing-today/queue/6794-4787-auto-f…

    ad0c7bb

  6. 12 remaining items

  7. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    Intake correction (fleet monitor, 2026-11-03)

    Correcting my Intake review posted minutes ago on this issue: its "no merged PR references this issue" line was wrong. The merged-PR verification for this run returned after the review was posted, and the issue's own 08:20Z update (which I also missed) already recorded the current state:

    Apologies for the noise; the enrichment labels and acceptance-criteria section stand, with this correction governing where they differ.

  8. github-actions
    assigned
    dezren39
    on Nov 3, 2026
  9. dezren39
    added
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026

  10. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    agent-note:v1
    state: triaged (coverage pass)
    Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as small_fix at priority 50 (priority/high). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
    Found via: coverage pass 2026-11-03

  11. dezren39
    added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    status/in-progressQueue worker lifecycle state
    and removed
    status/needs-triageNot yet triaged; needs a first look
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 4, 2026

  12. dezren39 commented on Nov 4, 2026

    @dezren39
    dezren39
    on Nov 4, 2026
    MemberAuthor

    agent-note:v1
    state: picked up
    task 7586 (queue repo-health-code-pilot, lane code-e) — small_fix revalidation of issue #4787. Prior state per issue comments: finding 2 resolved by PR #4797, finding 3 fixed by PR #4794; remaining live scope is finding 1 (the */15 schedule). Checking fresh schedule-run evidence, the 2026-11-03T17:26Z scheduled failure, and the AUTO_FORMAT_LAST_SUCCESS_SHA variable before deciding close vs fix.
    — repo-worker-code lane code-e, 2026-11-04T01:20Z

  13. dezren39 commented on Nov 4, 2026

    @dezren39
    dezren39
    on Nov 4, 2026
    MemberAuthor

    agent-note:v1
    state: done — closing with evidence (task 7586, repo-worker-code lane code-e, 2026-11-04)

    Decision recorded for finding 1: the */15 schedule IS firing — late/bunched

    Post-redesign schedule runs of auto-format.yml verified fresh today (the previous schedule run, 2026-11-02T16:33:52Z run 41482696862, was under the old design):

    • Run 41613820691 — 2026-11-03T09:47:02Z, schedule, success, head b7ee42302
    • Run 41684257818 — 2026-11-03T17:26:26Z, schedule, failure, head 4ab80c458 — it fired; the failure was a push-race rebase conflict at the push step (see follow-up below), not a scheduler or formatter failure
    • Run 41730853769 — 2026-11-03T22:26:44Z, schedule, success, head 00d7ef5d1

    So GitHub's scheduler delivers this workflow's */15 slots hours late and bunched, exactly the "late/bunched counts" case finding 1's acceptance names. No Actions-settings cause beyond that throttling was identified, and passes are landing: the 23:12Z pass pushed b65342bc ("style(treefmt): auto-format on main").

    All acceptance criteria verified

    Follow-up filed

    The one new defect found in this pass — the 17:26Z run's push-race rebase conflict aborting the pass indistinguishably from a formatter failure — is filed separately as #5230, per the one-issue-per-defect rule. It does not block closing #4787: the next scheduled pass recovered the range unaided.

    Closing #4787 as completed.

  14. dezren39
    closed this as completedon Nov 4, 2026
  15. dezren39
    removed
    status/in-progressQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/ciCI/CD pipelines, workflows, and build automation (area)effort/mMedium: expected under 1 day of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivesource/ci-watchFiled by the main-ci-failure-fix-trigger (fleet CI watch)task/small-fixQueue packet kind: bounded fix with reproduction and targeted tests

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions