Repository navigation
AA collector: a transient Supabase 403 wipes the explorer — keep last-good payload, skip query when batch unchanged #3332
Description
Observed live 2026-10-31 ~23:34 CDT: a fleet-feed collector tick got HTTP 403 from the Supabase Management API database/query endpoint and overwrote a healthy AA section (278 rankings, written by the 23:32 tick) with aaAbsent. The /aa explorer went empty/grey for the user (no providers selectable, points uncolored) until a manual rerun succeeded immediately — the 403 is transient/rate-limit-shaped (the same sbp_ management token serves several jobs, and the AA payload query is one giant jsonb build every 5 min).
Fixes, in pm-stack/aggregator/scripts/collect-fleet-feed.ts:
- Keep-last-good on failure. When the AA query fails, reuse the PREVIOUS feed's aa section verbatim, marked stale (keep its original generatedAt; set a stale/failure note in the section) instead of replacing it with aaAbsent. aaAbsent should mean 'never succeeded', not 'failed once'. Same treatment is worth auditing for the collector's other fail-soft sections.
- Skip the query when the batch hasn't changed. The AA payload only changes when ainventory.v_latest_workbook_batch changes. Do a cheap batch-id check first; if it equals the batch of the last successful payload build, reuse the cached payload and skip the giant query. This removes ~288 heavy Management API queries/day and most of the rate-limit pressure that causes (1).
Acceptance: kill the AA query artificially (bad ref in a test env) and confirm the feed keeps serving the last good aa section with a stale marker; confirm a tick with an unchanged batch performs no payload query (log line); explorer stays populated across a forced failure.
Activity
dezren39 commentedon Nov 1, 2026
Implemented directly and verified live 2026-11-01 (supersedes the queued task pkt3332 — it stands down).
PR #3349 (merged, fa29b8f): the collector stores the last good AA section at aa-last-good.json beside the feed, keyed by import batch + opencode-config fingerprint; a cheap batch probe reuses the stored section while batch+config are unchanged; any build failure degrades to the stored payload with stale:true + staleReason instead of blanking the section; absent now only occurs when no good build has ever succeeded. The same PR renamed the batch view usage to ainventory.v_latest_import_batch (new view added in Supabase; the old v_latest_workbook_batch is intentionally NOT dropped pending a consumer audit) and swept workbook->import terminology in the aggregator sources.
Live verification 2026-11-01 CDT:
Remaining AA flapping since is a separate fault — an unidentified stale-code writer overwriting the feed (code#3362) — not this issue's behavior: the deployed collector itself now degrades correctly.