Skip to content

fix(quota-plane): quota-gatherer failing every tick since 2026-11-04T18:45Z — auth.openCodeLocalCredentialSource is not a function #5638

Closed

Description

@dezren39

quota-gatherer.service failing every tick since 2026-11-04T18:45Z — import error auth.openCodeLocalCredentialSource is not a function

The 15-minute quota gatherer has produced no successful tick since
2026-11-04T18:30:15Z. Every tick from 18:45Z onward (18:45, 19:00,
19:15, plus extra manual/restart attempts at 19:02 and 19:22Z) fails
before any provider runs:

quota-gatherer: tick failed before providers ran: quota module import failed:
auth.openCodeLocalCredentialSource is not a function.
(In 'auth.openCodeLocalCredentialSource({ dbPath })',
'auth.openCodeLocalCredentialSource' is undefined)

Evidence (laptop amd-6, 2026-11-04 ~19:22Z):

  • quota-gatherer.service state: failed (Result: exit-code);
    a manual systemctl --user start quota-gatherer.service at 19:22Z
    by the pool-quota steward failed identically (deterministic code
    error, not a transient).
  • Heartbeat file ~/.local/state/pm-aggregator/quota-gatherer-heartbeat.json:
    {"tickAt": "2026-11-04T19:15:06.014Z", "ok": false, ...} with the
    same import error.
  • quota-current.json is frozen at generatedAt 2026-11-04T18:30:15.509Z
    (last good tick: 12 providers, 11 ok, 5,256 ms).
  • Timeline: the last good tick ran at checkout HEAD 45be0757b;
    the first failed tick (18:45Z) ran after HEAD advanced to include
    PR fix(aggregator): refresh stale quota on demand (#1171) #5263 (merge e25ebcdec8, "queue/6561-1171-quota-lazy-refresh").
    The lazy-refresh change is the prime suspect for the renamed/removed
    openCodeLocalCredentialSource export — verify against the diff.

Impact:

  • The PM board has fallen back to dated fallback windows for every
    pool (all rows state "stale"); the Turso hub leg no longer surfaces
    at all — the code#5468 storage escalation (87.866% of the 5 GB cap
    at the last good read, ~0.61 GB headroom) is invisible on the PM
    pages and its attention item is gone.
  • The pool-quota steward is proceeding on the 18:30Z read plus live
    OpenChamber /api/quota/* probes (per the standing rule, a stale
    read never idles a pool), but the hub picture — including the Turso
    leg, which has no live-probe route — cannot refresh until this is
    fixed.

Acceptance: quota-gatherer.timer ticks succeed again
(quota-current.json generatedAt advances, heartbeat ok: true),
and the board returns to pass-through values from the fresh read.

Found by: pool-quota-steward run 155 (2026-11-04T19:2xZ), step 6(1)
read-currency check. Related: #2490 (one quota read path), #5468
(Turso escalation currently unmeasurable), #3347 (Meta read — the
live-probe route still returns null windows independently of this).

Acceptance criteria

  • quota-gatherer.service completes a tick successfully: the auth.openCodeLocalCredentialSource is not a function import error is gone.
  • quota-current.json is regenerated by a scheduled tick (not only a manual run) and the heartbeat reports ok: true.
  • The regression source is identified (prime suspect: PR fix(aggregator): refresh stale quota on demand (#1171) #5263's lazy-refresh rename/removal of the openCodeLocalCredentialSource export) and the fix keeps the export surface the gatherer imports consistent with its consumers.

Activity

  1. developing-today commented on Nov 4, 2026

    @developing-today
    Contributor
  2. dezren39 commented on Nov 4, 2026

    @dezren39
    MemberAuthor

    Steward 156th run (2026-11-04T20:21Z): quota-gatherer still failing every tick — the 20:00Z and 20:15Z timer ticks and a manual systemctl --user start quota-gatherer.service at 20:21:06Z all fail identically: quota module import failed: auth.openCodeLocalCredentialSource is not a function. quota-current.json remains frozen at generatedAt 2026-11-04T18:30:15.509Z (~111 min stale at check); heartbeat ok:false. The Turso SaaS leg remains unmeasurable while the gatherer is down. The fleet proceeds on live OpenChamber /api/quota/* probes per the stale-read rule — no pool idled.

  3. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    priority/highImportant and time-sensitive
    effort/sSmall: expected under 2 hours of work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Nov 4, 2026
  4. dezren39 commented on Nov 4, 2026

    @dezren39
    MemberAuthor

    Intake review

    • Related: Prime suspect PR fix(aggregator): refresh stale quota on demand (#1171) #5263 (merge e25ebcd, queue/6561-1171-quota-lazy-refresh) — the first failed tick (18:45Z) ran right after HEAD advanced past it; the issue's own timeline lays this out. This outage also degrades fleet quota reads (the measurement gate runs on this data).
    • Notes: Deterministic code error, not a transient: every tick since 2026-11-04T18:45Z fails at module import (auth.openCodeLocalCredentialSource is not a function); quota-current.json is frozen at the 18:30:15Z tick. Filed with no labels; this pass normalizes them (kind/bug, priority/high, effort/s).
    • Risks: Until fixed, quota data is stale — dispatch decisions are being made on a frozen snapshot.
    • Suggested approach: Diff PR fix(aggregator): refresh stale quota on demand (#1171) #5263's auth export changes against the gatherer's imports; either restore the export under its expected name or update the import, then watch one scheduled tick go green.
      — fleet monitor intake, 2026-11-04
  5. dezren39 commented on Nov 4, 2026

    @dezren39
    MemberAuthor

    Pool-quota steward, 157th run (2026-11-04T21:26Z): STILL DOWN. The 21:00Z and 21:15Z timer ticks failed identically (auth.openCodeLocalCredentialSource is not a function), and a manual systemctl --user start quota-gatherer.service at 21:25:31Z failed the same way (heartbeat ok:false, durationMs 11). quota-current.json remains frozen at generatedAt 2026-11-04T18:30:15Z (~2h55m stale). Consequence this run: the hub picture incl. the Turso leg is unmeasurable; the steward live-probed every standard pool via laptop OpenChamber :3000 instead. New unverifiable-via-hub event: opencode-go popesdrew 5h hit 100% live (was 80.96% at 20:19Z) ahead of its 5:59 PM CST reset — the hub cannot confirm fleet-visible state until this is fixed. fleet-feed-collector, pm-aggregator and openchamber all healthy.

  6. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Pool-quota steward run 160 (2026-11-05T00:18Z) — post-tunnel-recovery confirmation: quota-gatherer.timer is ACTIVE and ticking (~12-min cadence; last tick 18:18:02 CST), but quota-gatherer.service still fails every tick in ~20ms with the same import error: 'quota module import failed: auth.openCodeLocalCredentialSource is not a function'. quota-current.json remains frozen at generatedAt 2026-11-04T18:30:15.509Z (~5h50m stale at check time); heartbeat file is fresh but ok:false with perProvider empty. fleet-feed-collector.timer active and fleet-feed.json fresh. A restart cannot fix a deterministic import failure — this needs the code fix; the steward proceeded on live OpenChamber /api/quota probes per doctrine (a stale read never idles a pool).

  7. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Steward note, 2026-11-05T01:20Z (161st run): the systemd unit still fails every tick at the same import error (auth.openCodeLocalCredentialSource is not a function; journal ticks 18:18–19:19 CST all failed; standalone heartbeat tickAt 2026-11-05T01:19:33Z ok:false, perProvider empty).

    New diagnostic fact: quota-current.json nevertheless ADVANCED — generatedAt 2026-11-05T00:27:47.702Z, embedded heartbeat ok:true with per-provider rows (incl. turso via turso-api) — ~6h newer than the previously frozen 2026-11-04T18:30:15Z read. The successful 00:27:47Z write did not come from a timer tick (the 18:30:13 CST tick failed; file mtime is 18:27 CST), so a second writer/refresh path landed one good gather while the unit stays broken. The read is again aging (52 min old at this run's fetch; >20-min freshness gate), so the unit failure still stands as the blocker for a current read.

  8. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Steward 162nd run (2026-11-05T02:22Z): the quota-gatherer has RECOVERED. quota-gatherer.service ticks at 2026-11-05T02:15:02Z and 02:21:29Z both completed status 0/SUCCESS (providers=12 ok=10, durationMs ~6.8-8.4s); quota-current.json generatedAt advanced to 2026-11-05T02:21:29.564Z with heartbeat ok:true. The auth.openCodeLocalCredentialSource is not a function import failure that failed every tick since 2026-11-04T18:45Z no longer occurs (laptop primary ~/git/code advanced to HEAD 83f78ed, clean). Remaining per-tick errors are provider-side only: claude 'Rate limited. Retrying soon.' and meta no-data (null windows, retained last-known from 2026-11-04T18:30Z; tracked on #3347). Steward did not restart anything — recovery arrived via the code path. This issue's failure mode appears resolved; closing is the fix owner's call.

  9. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    on Nov 5, 2026
  10. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Agent note - handed off

  11. added
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 5, 2026
  12. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Triage — queue task 10093

    The operational failure is resolved. Steward run 162 records scheduled quota-gatherer.service ticks succeeding at 2026-11-05T02:15:02Z and 02:21:29Z, quota-current.json advancing to 2026-11-05T02:21:29.564Z, and heartbeat ok:true (recovery evidence).

    Current main is f0fa32003fe5399ff6bc117d08ecc80a7322250d. The durable code path in PR #6136 (change 97f8871b8b201f658cafba00bf2fd327272e862a, merged as 97cd55c) replaces the gatherer's in-process OpenChamber auth/quota import with opencode quota --json; the missing openCodeLocalCredentialSource export is no longer called. The suspected PR #5263 did not touch the gatherer/auth import and merged after the reported first failure. The runtime recovery was already reported at checkout 83f78ed2347e, before PR #6136; repository history does not explain that immediate recovery, so its cause remains unconfirmed.

    The remaining Claude rate-limit and Meta no-data conditions are provider-side and tracked separately on #3347. Since scheduled ticks recovered and current main removes the failing dependency, closing #5638 as completed; no blocker remains for this failure mode.

  13. removed
    task/triageQueue packet kind: verify, reproduce, and route this item
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivequeue/claimedQueue worker lifecycle statestatus/in-progressQueue worker lifecycle statestatus:donework/assignedQueue worker lifecycle state

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions