Repository navigation
fleet(queue): queue control plane down - PostgREST 400 then 503 PGRST002 / 521, verbs after claim blocked #2768
Description
Problem
The repo-health queue control plane (Supabase PostgREST backing ~/tools/repo-queue.py) is down for all lanes, blocking every queue verb after claim-time:
- ~14:05Z 2026-10-30:
repo-queue.py health --queue repo-health-code-pilot→ HTTP 400 from PostgREST (empty message), twice.claimstill succeeded at 14:06Z. - ~14:15Z onward:
complete,heartbeat, andhealth→ HTTP 503PGRST002("Could not query the database for the schema cache. Retrying."), repeatedly over ~15 minutes; one attempt returned a Cloudflare 521 "Web server is down" page for supabase.co.
Impact
- Task 3572 (impl(aggregator+collector) part 1 of 2: Project pool graph edges contract (split of #1122) #2399) finished its GitHub work — PR pm-stack: Project pool graph edges contract (#2399, part 1 of #1122) #2744 merged (111a318), issue closed, labels cleaned, backlog + sweep.log updated — but its terminal
completeverb cannot land while the backend is down. Its lease (to 14:51Z) cannot be heartbeated either, so the task will fall back to queued and need a resume-claim whose only remaining step is the closing verb (state is fully checkpointed on the task and on issue impl(aggregator+collector) part 1 of 2: Project pool graph edges contract (split of #1122) #2399). - All other lanes holding claims face the same evaporation risk through no fault of their runs; the fleet monitor's hourly divergence repair is the backstop.
Acceptance
repo-queue.py health --queue repo-health-code-pilotreturns counts again (no 400/503).- The pending
completefor task 3572 lands (by the original lane or a resume claim), and any other claims stranded by the outage are reconciled (completed from their checkpoints/GitHub state, not re-worked). - Root cause noted here (PostgREST schema-cache / upstream availability) with the outage window.
Activity
dezren39 commentedon Oct 30, 2026
Lane A update 14:44Z: task 3572's
completeretried 13+ times between ~14:15Z and 14:44Z — every attempt HTTP 503 PGRST002 (one 521). Lease expires 14:51Z with the GitHub work fully done (PR #2744 merged 111a318, #2399 closed, checkpoints on the task through awaiting_ci, done agent note on #2399, backlog + sweep.log updated). On backend recovery, the first claim of 3572 (or monitor reconciliation) only needs the closingcompleteverb with outcome fixed_merged / merge_sha 111a318 — do not re-work the PR.dezren39 commentedon Oct 30, 2026
OpenChamber scheduled lane (worker
repo-worker-code, queuerepo-health-code-pilot) update — still down at 15:44Z.healthcall failed HTTP 503PGRST002("Could not query the database for the schema cache. Retrying."), one HTTP 522 Cloudflare page at ~15:18Z. Zero successful queue reads or writes in ~65 minutes.claim --worker repo-worker-code --lease-seconds 2700at ~14:46Z failed the same way, so this lane holds no claim and is leaving nothing stranded — no claim, no checkpoint, nothing to reconcile on my side.https://nltgxorqwxhgfzseiirt.supabase.co/rest/v1/returns 401 at the root (edge reachable, auth layer alive) while PostgREST behind it 503s on the schema cache. Supabase's public status page still reports "All Systems Operational", so this is project-scoped, not platform-wide — consistent with a stuck schema-cache reload rather than an origin outage.Resume condition unchanged: one successful
repo-queue.py healthforrepo-health-code-pilot. Given ~65 minutes of continuous 503s, a project-level intervention (PostgREST restart / schema cache reload via the Supabase dashboard, or the pending task 3572completefrom the lane-A note) is likely needed rather than more retries.dezren39 commentedon Oct 30, 2026
Intake review
🤖 Agent note · handed off
Still live? Yes — verified this run at ~16:12Z:
repo-queue.py healthreturned HTTP 503 PGRST002 ("Could not query the database for the schema cache") for ALL FOUR pilot queues (code, opencode, openchamber, nix). The issue's own## Acceptancesection stands as its acceptance criteria, so no duplicate section was appended.Notes / risks: #2767 covers the same outage window from a second run's evidence — the two should be reconciled (one kept as the canonical outage record) once recovery lands, not before. Until health returns, every queue verb (enqueue/claim/park/complete/cancel/bind) is blocked fleet-wide; GitHub labels are the only truthful state surface in the meantime, and claim leases will keep lapsing back to queued — that is expected outage behaviour, not worker evaporation.
Suggested approach: on recovery — post the recovery timestamp + cause here, land stranded terminal verbs from checkpoints, then run the monitor's divergence repair across all queues.
dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · done (triage)
Triage verdict (task 3957): NOT CONFIRMED — control plane recovered, closing as resolved
The outage does not hold. This run (2026-11-03 ~00:40–00:55Z) transacted against the queue control plane continuously with zero PostgREST/Cloudflare errors: 4 claims (tasks 2881, 6298, 3610, 3957), ~10 checkpoints, 1 enqueue (follow-up task 6667), 1 park, 3 completes — every verb returned 0. The 2026-10-30 503 PGRST002 / 521 window described in the body is over; recovery happened sometime between the last 503 sighting (10-30 ~16:12Z per the monitor intake) and this run.
Routing: no follow-up packet — there is nothing to fix in fleet code (the outage was upstream Supabase/PostgREST health, as the intake already assessed). Closing this issue as resolved-with-evidence. Companion #2767 covers the same window and is currently held by task 3956 in another lane; its holder verifies independently — not closed from here.