Repository navigation
pm-aggregator /aa explorer data feed broken: Supabase query 403 after muse-dev2 migration #3294
Description
Live check 2026-10-31 21:09 CDT: curl localhost:4870/api/aa.json on the laptop aggregator returns {"status": "absent", "reason": "Supabase query failed (HTTP 403)"} — the AA Explorer / Model Explorer tabs have no data.
Context: the AA measurements DB moved to Supabase project muse-dev2 (schema ainventory, AA v5: 140 rows / 138 scored, live). The aggregator's read path still fails against it — consistent with the known RLS posture (RLS enabled, zero policies; anon/publishable keys read nothing; reads must use a privileged server-side path). Likely the collector/feed still points at the old muse-dev project ref or a key without access on muse-dev2.
Fix: point the aggregator's AA feed at muse-dev2 ainventory via the privileged read path (service key from the vault by name, never in the repo), verify /api/aa.json returns status ok with v5 counts (rankings ~140), and confirm both explorer tabs render the v5 data.
Activity
dezren39 commentedon Nov 1, 2026
Root causes found + live fixes applied 2026-10-31 evening (main chat):
AA feed 403: the fleet-feed collector (sandbox, fleet-feed-collector.service → pm-feed checkout → collect-fleet-feed.ts) defaulted SUPABASE_PROJECT_REF to the retired muse-dev ref nltgxorqwxhgfzseiirt. After the muse-dev2 migration every AA database/query hit the dead project → 403 → aaAbsent. LIVE FIX: systemd drop-in on the sandbox setting SUPABASE_PROJECT_REF to the muse-dev2 ref (Bitwarden supabase.project-ref). Verified: collector run produced 'aa 278 rankings / 35 configs / 183 test runs', feed pushed, laptop /api/aa.json status ok with v5 data. The sbp_ management token (supabase.access-token-muse-cli) works against muse-dev2 unchanged. DURABLE REMAINDER: change the default ref in collect-fleet-feed.ts (or source it from bws in the collector) — the /etc/systemd drop-in does not survive sandbox recycles.
Usage explorer stale (same family): the aggregator renders its curated quota rows from quota-view.md at ~/workspace/goals/stewardship-.../visibility/quota-view.md resolved on the LAPTOP — that file has never existed on the laptop. quota-check rewrites it every run on the SANDBOX; nothing syncs it across, while the live pool windows (quota-current.json, gatherer 15-min) were wired correctly all along. LIVE FIX: copied the current quota-view.md to the laptop ~/.local/state/pm-aggregator/quota-view.md, set PM_AGG_QUOTA_VIEW there in the aggregator .env, restarted pm-aggregator; Go rows now carry tonight's extracted prose. DURABLE REMAINDER: a recurring sync — e.g. quota-check pushes quota-view.md alongside its update, or the fleet-feed push carries it.
dezren39 commentedon Nov 1, 2026
Collector migrated to the laptop — it no longer runs on the sandbox (Drew's order 2026-10-31 21:29 CDT: laptop-only execution; the sandbox is orchestration only).
New home: laptop USER units
fleet-feed-collector.service/fleet-feed-collector.timer(~/.config/systemd/user/), runningbun run scripts/collect-fleet-feed.tsfrom the primary checkout ~/git/code/pm-stack/aggregator every ~5 min (OnBootSec=2min, OnUnitActiveSec=5min), writing the feed DIRECTLY to ~/.local/state/pm-aggregator/fleet-feed.json — the aggregator's existing PM_AGG_FEED_FILE path (the old scp destination ~/pm-stack/aggregator is a symlink to that state dir, so the reader side is unchanged; no proxy/scp machinery anymore).Env in the unit: SUPABASE_PROJECT_REF set to the muse-dev2 ref (fetched from Bitwarden
supabase.project-refon the laptop); the collector's bws resolution works natively on the laptop via BWS_BIN=/run/current-system/sw/bin/bws + BWS_ACCESS_TOKEN_FILE=~/.config/bws/access-token (bothsupabase.access-token-muse-cliandgithub.project-tokenresolve). QUOTA_VIEW_PATH points at the laptop state copy of quota-view.md.Verified live: manual run + one full timer cycle (21:37 CDT) both wrote the feed; aa section status ok, 278 rankings / 35 configs / 183 test runs; /api/aa.json returns status ok with 278 rankings; GitHub Projects section populates (343 items) from the laptop.
Sandbox retired: fleet-feed-collector.timer disabled + stopped, service stopped, and tonight's /etc/systemd/system/fleet-feed-collector.service.d/override.conf drop-in removed. Verified: timer disabled/inactive, zero fleet-feed timers on the sandbox.
Known fail-soft losses (sections omitted, nothing errors): inputs that live only in the sandbox workspace no longer reach the collector — sandbox OpenCode sessions + Jules state, orchestration.json, the DNS register, provider-quota-registry records, charges candidates, and the usage-snapshot outbox. Those sections render absent until/unless laptop-side producers or a sync exist. The aggregator's own laptop-native inputs (quota-current.json etc.) are untouched.
Durable remainders still standing: (1) the collector's in-code default SUPABASE_PROJECT_REF is still the retired muse-dev ref — the laptop unit env now carries the muse-dev2 ref permanently, but the code default should still be fixed (or sourced from bws) in a normal PR; (2) the laptop quota-view.md copy is still a one-time copy from the earlier fix — its recurring sync is unchanged by this migration.
fix(collector): home-host session label + Supabase ref resolves to mu…
Merge pull request #3301 from developing-today/feed/collector-laptop-env
dezren39 commentedon Nov 1, 2026
TASK A complete — every fleet-feed input re-homed to the laptop (2026-10-31 evening)
The collector (laptop user units
fleet-feed-collector.service/.timer) now reads nothing from sandbox-local files. Per-section ledger:jules-review-state.jsonin the sandbox goal workspaceJULES_STATE_PATH→ the laptop's hatch-personal clone copy (same git file; the clone is kept current bylaptop-autocommit-hatch-personal.timer)julesSummarypresentservice-prod.json); password extracted per-run byExecStartPre(~/.local/bin/fleet-feed-oc-password.sh, same source oc-password-sync uses) + Julesorchestration.jsonin the sandbox goal workspaceORCHESTRATION_PATH→ laptop hatch-personal clone copyorch-2026-10-31T19:16:28Z, served at/api/orchestration.jsonDNS_REGISTER_PATH→ laptop hatch-personal clone copyCHARGE_CANDIDATES_PATH/CHARGES_SURFACED_STATE_PATH→ laptop hatch-personal clone copiesQUOTA_REGISTRY_PATH→ laptop hatch-personal clone copyQUOTA_VIEW_PATH→ laptop hatch-personal clone copy (md5-identical to the sandbox's current file; quota-check's future updates flow via git)USAGE_SNAPSHOT_OUTBOX_PATH→ laptop-local~/.local/state/pm-aggregator/usage-snapshot-outbox.jsonlRoute notes:
CLAUDE_LAUNCH_RECORDS_PATHpoints at the clone path so records appear if the file ever lands in git.Quota pools: 5 is correct — the old "11" was a stale-checkout artifact
The pre-migration 11 pools came from the sandbox's stale pm-feed checkout: old collector code + an opencode.jsonc whose
disabled_providers(18 entries) predates the current deployed config (24: addsmeta,deepseek,commandcode,antigravity-ls,antigravity-agy,muse). Against the current code + deployed config, #1114 filters exactly the 6 pools whose every door is disabled (meta, claude-pro, commandcode, antigravity×2, deepseek) — the collector logs each filter line. No laptop input is missing; restoring those rows would mean defeating #1114. If walled pools should stay visible on the quota page, that's a #1114 policy change — Drew's call.Code change: PR #3301 (merged, deployed)
fix(collector): home-host session label + Supabase ref resolves to muse-dev2— the durable remainder from this issue's first comment is now in code:SUPABASE_PROJECT_REFresolves env → Bitwardensupabase.project-ref→ muse-dev2 fallback (never the retired muse-dev ref), so neither the sandbox drop-in nor the laptop unit env is load-bearing anymore. It also addsOPENCODE_SESSION_SOURCE(the feed now truthfully labels the laptop server's sessionsopencode-laptop—SessionSourcealready carried the value) andOPENCODE_SERVER_URLahead of the legacy alias. The laptop primary pulled the merge commit (37f9590) via pm-git-sync; live feed verified after a unit-driven run: attention = 60opencode-laptop+ 28jules, registry 88, checkout in-sync.Aggregator test suite on the PR branch: 731 pass / 7 fail — the identical 7 fail on pristine origin/main (pages-manifest/overview/aa-explorer registration tests; pre-existing, unrelated to the collector — they line up with the "explorer is missing views" report and are flagged for the parity work).
Primary-clone note: pm-git-sync failed loudly once (21:54) over three untracked backups in ~/git/code —
.opencode/opencode.jsonc.bak-2026-10-31-{agy-out-primary,commandcode-off,primary-gpt}— left untouched per the primary-clone rule; the primary subsequently reached the merge commit and reports in-sync.dezren39 commentedon Nov 1, 2026
Root cause found and fixed via #3362 / PR #3865: the 403 was not the laptop aggregator's read path (healthy: muse-dev2 + bws management token returns the full ainventory payload, HTTP 201) — it was a stale sandbox-side collector (
~/git/pm-feedclone, retired muse-dev project ref default, muse-dev2-scoped token -> 403) pushing an AA-absent feed over the laptop's good feed every ~5 min. Sandbox collector now retired;/api/aa.jsonverifiedstatus: ok(batch fe02783b, 278 rankings). Evidence + timestamps on #3362.(This comment replaces an earlier version whose inline-code spans were stripped by shell quoting during posting.)
developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 30 (priority/none). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03