Repository navigation
fix(quota-plane): Meta quota read path dead on both routes — hub file-export 'no-console-page' (16 ticks) + live /api/quota/meta null windows (20 probes) #3347
Description
Problem
The Meta (Muse API High Usage) pool has had no usable quota read on either
route for 16+ consecutive steward runs, and the failure will still be live
when the pool's weekly window resets Sun Nov 1 2026 18:00 CST — the
moment a fresh read is most needed to judge the reopening.
Evidence (pool-quota-steward, snapshot-first runs per #2135)
- Hub file-export route:
quota-current.json(laptop pm-aggregator,
15-min tick) has carried meta stateerror—
file-export status: no-console-page— for 16 consecutive copied
ticks, latest @2026-11-01T05:00:42Z (generatedAt 2026-11-01T05:15:14Z
heartbeat itself ok; every other ok pool in the same file is fresh). - Live route: the laptop OpenChamber
GET /api/quota/metareturns
ok:truewith null window percentages (rolling + weekly) for 20
consecutive steward probes, latest 2026-11-01 ~00:2x CDT
(raw: goal hidden_files/dispatch-packets/quota-2026-11-01-0017-live.txt). - Last precise Meta read: 2026-10-31T13:01:13Z (weekly 100%, the read
the pool's standing closure rests on). Every figure since is unverified. - The exporter built by feat(quota-plane): Meta adapter Phase 1 — laptop-side quota exporter (subscription-quota API, 15-min file export) #1301 (CLOSED) is the component whose export now
fails; meta-exporter: CDP plumbing has no timeout — an endpoint that accepts but never answers hangs the exporter forever and writes no failure document #2175 (CDP hang) is a different failure shape and does not cover
this one. Claude's read path errors identically on both routes
("Rate limited. Retrying soon.") and may share a cause — worth a look
while in there, not required.
Impact
- The steward's measurement gate cannot verify the Meta weekly figure;
the pool stays decidable-closed only on the last precise read + rule,
with compliance judged on the seating prong alone (zero Meta-provider
sessions across every activity map in the span — that prong held). - At the Nov 1 18:00 CST reset, there is no read path to confirm the
window actually reset or to seat the pool again on evidence.
Acceptance
quota-current.jsoncarries a fresh ok meta entry with real window
percentages for 2 consecutive ticks, OR the export failure is diagnosed
to a named root cause with its own follow-up issue.GET /api/quota/metaon the laptop returns non-null window percentages
agreeing with the hub entry.- The first steward run after a fix records the recovered read in
hidden_files/pool-change-ledger.md and closes the read-path follow-up
item carried since the 2026-10-31 11:24Z run.
Filed by the pool-quota-steward daily pass, 2026-11-01 (the judgment the
2026-10-31 daily pass deferred to today: whether this needs a fix issue
beyond the standing ledger record — it does).
Acceptance criteria
- At least one Meta quota read route returns real window percentages again: the hub file-export no longer reports
no-console-pagefor meta, or the live/api/quota/metareturns non-null rolling + weekly windows — verified on consecutive probes, not a single lucky read. - The root cause of each dead route is named on this issue (what changed, where), not just the symptom clearing.
- A fresh, precise Meta read exists before the weekly window reset (Sun 2026-11-01 18:00 CST) so the reopening can be judged from data.
Metadata
Metadata
Assignees
Labels
pm-stack (Pad/Grist) in developing-today/codeMedium: expected under 1 day of workSomething isn't working (prefixed form of the classic bug label)Intake review done: acceptance criteria, links, and normalized labels addedImportant and time-sensitiveFiled by the new-issue-creator jobQueue packet kind: verify, reproduce, and route this item
Activity
dezren39 commentedon Nov 1, 2026
Intake review
Related: openchamber#894 (the /api/quota/* credential-path failure — same measurement plane, different surface), #2135 (pool quota steward protocol), code#3294-lineage AA/quota read fixes. Notes: time-critical — the Meta weekly window resets today 18:00 CST and the standing closure rests on the 2026-10-31T13:01Z read. Two independent routes are dead with different failure shapes (file-export no-console-page vs live null windows), which suggests two causes, not one. Suggested approach: treat the routes separately; the file-export route is a console-page/session problem on the laptop aggregator, the live route is an upstream response-shape problem — verify each against its own evidence file before changing anything.
— Fleet monitor intake pass (protocol §5), 2026-11-01. Still live at review time: no merged PR or commit found that resolves this item (checked against the issue's own evidence and recency).
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7385 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
24 remaining items
dezren39 commentedon Nov 4, 2026
Pool-quota steward refresh — 2026-11-04T10:22Z (146th run): the Meta read regression persists a TWELFTH consecutive tick set. Hub quota-current.json generatedAt 2026-11-04T10:15:16Z (source openchamber-quota, heartbeat ok) returns the meta entry state ok with NULL rolling/weekly windows (measuredAt 10:15:20Z), and the live laptop /api/quota/meta probe minutes later returns ok:true with the same null windows. Last verified real percentages remain 2026-11-03T22:45Z: 5h 4% / weekly 100%. The steward continues to proceed on the pool as CLOSED on the weekly until Sun Nov 8 6:00 PM CST (per the failed/stale-read rule, the pool is not idled — dispatch proceeds on last known state). No change in failure mode this tick set.
dezren39 commentedon Nov 4, 2026
Steward 147th run, 2026-11-04T11:21Z: Meta read regression persists a thirteenth consecutive tick set. Hub quota-current.json generatedAt 2026-11-04T11:15:09Z (heartbeat ok, meta state ok) returns NULL rolling/weekly windows (measuredAt 11:15:21Z), and the live laptop /api/quota/meta probe at ~11:21Z agrees — ok:true with null windows. Last verified real read remains 2026-11-03T22:45Z (5h 4% / weekly 100%); the steward continues to proceed on that as CLOSED on the weekly until Sun Nov 8 6:00 PM CST. No PAYG movement (live deepseek ¥44.29 flat).
dezren39 commentedon Nov 4, 2026
Steward refresh — 2026-11-04T12:30Z (148th run): the Meta quota read regression persists a FOURTEENTH consecutive tick set. Hub quota-current.json (generatedAt 2026-11-04T12:15:13Z, heartbeat ok) and the OpenChamber read both return state ok with NULL rolling/weekly windows for meta (measuredAt 12:15:20Z). Live per-provider probes this run mostly transport-failed because the OpenCode backend is mid-restart on the laptop (session/active returns 503 "OpenCode is restarting"), so the fresh hub read is the picture of record. Last verified Meta windows remain 2026-11-03T22:45Z: 5h 4% / weekly 100% — the pool continues to be proceeded-on as CLOSED on the weekly until Sun Nov 8 6:00 PM CST. No config change made by the steward.
dezren39 commentedon Nov 4, 2026
Steward refresh — 2026-11-04T13:30Z (149th run): the Meta quota read regression persists a FIFTEENTH consecutive tick set. Hub quota-current.json (generatedAt 2026-11-04T13:15:08Z, heartbeat ok, tick 7,344 ms) returns state ok with NULL rolling/weekly windows for meta (measuredAt 13:15:15Z), and the live laptop OpenChamber probe (http://100.99.113.103:3000/api/quota/meta, ~13:27Z, after OpenChamber's restart by others at 07:23:45 CST) agrees exactly: ok:true with usedPercent null on both windows. The pool stays proceeded-on as CLOSED on the weekly (last verified 2026-11-03T22:45Z: 5h 4% / weekly 100%, reset Sun Nov 8 6:00 PM CST) per the failed-read doctrine — no placement depends on a fresh Meta number. Board side: /api/quota.json still renders the stale dated fallback (weekly 98%, state "near", attention 98%) against the null read — tracked on #5101, refreshed separately this run. Next check: the 2026-11-05 daily steward pass.
dezren39 commentedon Nov 4, 2026
Steward 150th run (2026-11-04T14:32Z) — Meta read regression persists, SIXTEENTH consecutive tick set: hub quota-current.json generatedAt 2026-11-04T14:15:10Z reports state ok with rolling and weekly windows both null, and the live laptop probe (OpenChamber /api/quota/meta, ~14:30Z) agrees exactly (ok:true, both windows null). Last verified read remains 2026-11-03T22:45Z (5h 4% / weekly 100%); the pool continues to be proceeded-on as CLOSED on the weekly until Sun Nov 8 6:00 PM CST. No placement depends on a fresh Meta read this run (a failed/stale read never idles dispatch on other pools).
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 151st run (2026-11-04T15:32Z): the Meta read regression persists a SEVENTEENTH consecutive tick set. Hub quota-current.json generatedAt 2026-11-04T15:15:02Z (source openchamber-quota, heartbeat ok) returns state ok with rolling AND weekly windows null; the live laptop OpenChamber /api/quota/meta probe at ~15:24Z agrees exactly (both windows null). Pool remains proceeded-on last verified (2026-11-03T22:45Z tick: weekly 100%) as CLOSED until Sun Nov 8 6:00 PM CST per the failed-read doctrine — no placement depends on a fresh Meta read.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 152nd run (2026-11-04T16:24Z): the Meta quota read regression persists — an EIGHTEENTH consecutive tick set. Hub quota-current.json generatedAt 2026-11-04T16:15:02Z (source openchamber-quota, heartbeat ok) reports meta state ok with rolling AND weekly windows null; the live laptop OpenChamber /api/quota/meta probe agrees exactly (both windows null). Last verified read remains 2026-11-03T22:45Z (5h 4% / weekly 100%); the steward continues to proceed on that as CLOSED on the weekly until Sun Nov 8 6:00 PM CST. A failed/stale read never idles the pool beyond its last verified state; no placement depends on a fresh Meta figure this run.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 153rd run (2026-11-04T17:24Z): the Meta read regression persists into a NINETEENTH consecutive tick set.
quota-current.jsongeneratedAt 2026-11-04T17:15:02Z (gatherer heartbeat ok): themetaentry is state ok with rolling AND weekly windows null (measuredAt 17:15:13Z)./api/quota/meta, ~17:20Z) agrees exactly: ok:true, both windows' usedPercent null.The pool remains proceeded-on as CLOSED on its last verified state (2026-11-03T22:45Z tick: weekly 100%, until Sun Nov 8 6:00 PM CST). No placement depends on a fresh Meta read; a stale/failed read does not idle the pool per standing doctrine. Next daily steward pass (2026-11-05) judges recovery vs continued regression.
dezren39 commentedon Nov 4, 2026
Steward run 154 (2026-11-04T18:24Z): Meta read regression persists — twentieth tick set. Hub quota-current.json generatedAt 2026-11-04T18:15:03Z: meta state ok with rolling/weekly windows NULL (measuredAt 18:15:06Z); the live laptop probe (OpenChamber /api/quota/meta) agrees exactly — both windows null. Pool remains proceeded-on as CLOSED on the weekly (last verified 2026-11-03T22:45Z: 5h 4% / weekly 100%; weekly resets Sun Nov 8 6:00 PM CST). No placement this run depends on a fresh Meta read.
dezren39 commentedon Nov 4, 2026
Steward run 155 (2026-11-04T19:22Z): the Meta read regression persists.
/api/quota/metaat ~19:22Z):ok: truewith rolling + weekly windows still null — the liveroute agrees with the hub exactly, as in every prior probe.
quota-current.json)also carries null Meta windows. No newer hub tick exists: the
gatherer itself has been failing every tick since 18:45Z on an
unrelated import error — filed separately as fix(quota-plane): quota-gatherer failing every tick since 2026-11-04T18:45Z — auth.openCodeLocalCredentialSource is not a function #5638.
2026-11-03T22:45Z: weekly 100%, until Sun Nov 8 6:00 PM CST). No
placement depends on a fresh Meta read this run.
dezren39 commentedon Nov 4, 2026
Steward 156th run (2026-11-04): Meta live probe via laptop OpenChamber /api/quota/meta at ~20:19Z returns null rolling and weekly windows again — 22nd consecutive confirmation of the read regression. Hub ticks remain suspended by the gatherer failure (#5638); the last good hub tick (18:30:15Z) also read null. Pool proceeded-on last verified (2026-11-03T22:45Z tick: weekly 100%) as CLOSED until Sun Nov 8 6:00 PM CST.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 157th run (2026-11-04T21:26Z): Meta read still null — 23rd consecutive confirmation. Live probe via laptop OpenChamber :3000 /api/quota/meta at ~21:20Z returned ok:true with rolling and weekly windows both null (no percentages, no resets); the frozen hub entry (18:30:15Z, suspended by the gatherer failure on #5638) agrees. Pool continues to be proceeded-on as CLOSED on the last verified read (2026-11-03T22:45Z: weekly 100%, until Sun Nov 8 6:00 PM CST). DeepSeek PAYG balance read live in the same probe set: ¥44.29, flat — no PAYG movement.
dezren39 commentedon Nov 5, 2026
Determination — why
subs_usageis null (investigation 2026-11-04 ~23:25–23:50 CST, at Drew's direction)Short answer: Meta's server is not serving a usage object for this subscription right now, to any client — including its own CLI. The current null coincides with the subscription sitting at its wall: a live inference attempt on the OAuth plan route just returned
429 "Subscription quota exhausted. Your usage window resets at 2026-11-09T00:00:00Z."The key endpoint keeps returning the subscription metadata (active, tier) and omitssubs_usageentirely in this state. Our fetcher is faithful; the null is Meta's answer. Drew's hypothesis — "maybe it goes null when it hits 100%" — is the best-supported reading of the current episode, with one historical correction below.Evidence (all tonight, laptop, credentials by name only)
muse execwithMETA_API_KEYstripped (OAuth plan route):API error 429: Subscription quota exhausted. Your usage window resets at 2026-11-09T00:00:00Z.— the exact reset anchor of record.POST https://api.meta.ai/muse-code/key,x-api-version: 1.0.0, body{"onboard": false}, Muse account OAuth token from~/.config/muse/auth.json): HTTP 200,is_subs_active=true,subs_tier_name="Muse Code High Usage",require_payment=false— and the response's key set contains nosubs_usageat all (not a present-null; the field is absent). Same result withonboard: trueand with an empty body.muse usage --jsonin OAuth mode (env key stripped) returns{"allowance": null, "reason": "server_sent_no_usage", "hint": "the server sent no usage data for this subscription; this can happen before your first use in a new window"}. Two first-party surfaces, same answer: no usage object is being published. Caution: the CLI's hint text is a generic client guess and is factually wrong for the current state — the window is exhausted, not unused. Do not let it steer anyone again.meta.muse-code-plan.api-key→ 401; BWSmeta.muse-api-default-project.api-key→ 401. Drew's "maybe the API key gets usage" — tested, no. The API key cannot fix this read.no-data, row retained per fix(quota-gatherer): ok-but-null retains last-known (no-data); Command Code explicit ask + plan windows (#5704) #5719). The null began within ~90 minutes of the weekly reaching 100% and has persisted at the wall for 29+ hours.The historical correction (why "null ⟺ 100%" is not the whole law)
The live route also returned null for 51 consecutive probes across 2026-10-28→11-02, a span where the console showed weekly climbing ~9% → 100%, and it was still null immediately after the Nov 2 reset at a console-verified 0%. So the key endpoint has omitted
subs_usageat low usage and on a fresh window too — its usage publication is unreliable in both directions, and that earlier span's trigger is not determinable from today's evidence (it coincides with the post-PAYG-incident period and the fetcher's first days). What is proven: the key endpoint cannot be the pool's system of record for percentages — in any state.The working read (real percentages)
The dev.meta.ai portal API (cookie-auth, laptop Helium session) served exact percentages whenever it was read:
GET https://dev.meta.ai/api/portal/teams/1073226932208501/subscription-quota→{subscription_quota: {tier, window_weighted_used/limit, window_resets_at, weekly_weighted_used/limit, weekly_resets_at}}(proven live payload 2026-10-27: window 1.02%, weekly 8.83%, resets matching the key endpoint's own window boundaries), plusGET .../billing-bannerfor remaining free credits ($148.57). This was the #1301 exporter's source and the hub's governing read for weeks. Today's blocker for this route is only the cookie: the Oct-era profile copy'sllama_dev_sessnow returns 401 (session rotated); the live session sits in the desktop Helium's default profile, which currently has no debug port and was occupied by other work during this investigation. The fix — a laptop-side portal fetcher for the gatherer, with CDP cookie capture from the live profile — is filed as a follow-up issue (linked below) with the full recipe; this issue's root-cause acceptance is met for both routes: the live route's cause is this determination, and the hub route's cause was the #5040 source switch onto this same null key-endpoint read (the console source it replaced was retired while still the only faithful one).State left unchanged: pool remains proceeded-on as CLOSED (weekly 100%, resets 2026-11-09T00:00Z) per the failed-read doctrine — now corroborated by the server's own 429.
dezren39 commentedon Nov 5, 2026
Follow-up filed: #6079 — restore the portal-API fetcher (subscription-quota) as the gatherer's Meta percentage source, with the full recipe and the live-cookie capture step. The key-endpoint read stays as the active/tier cross-check.
dezren39 commentedon Nov 5, 2026
Pool-quota steward daily pass 2026-11-05 (166th run, ~06:15-06:30Z): Meta read regression persists — 30th consecutive null confirmation. Hub quota-current.json generatedAt 2026-11-05T06:15:18Z retains the stale entry @2026-11-04T18:30:20Z with rolling/weekly windows null (retainedFromFailedTick set); live laptop OpenChamber /api/quota/meta at ~06:22Z also returns null windows. Pool proceeded-on as CLOSED on last verified weekly 100% (2026-11-03T22:45Z) until Sun Nov 8 6:00 PM CST per the failed-read doctrine. Daily-pass verdict: REGRESSION (unchanged); follow-up stays open, re-judged at the 2026-11-06 pass.