Repository navigation
epic(fleet): queue fleet reliability — lane workers for every queue, certified shift reporting #1233
Description
Queue fleet reliability
Purpose
The queue system that keeps the four repos healthy (Supabase agent.tasks, the repo-health queues, lane shifts, the fleet monitor) proved itself on 2026-10-26 — and showed exactly two reliability defects. This epic closes both, so the fleet runs for a week at a time without operator repairs.
Scope
- Claim evaporation on the non-code queues. The code queue runs two staggered lane shifts (~90-min budgets); opencode/openchamber/nix are worked by short-budget arrangements whose runs end mid-task, stranding claims. On 2026-10-26 the monitor repaired 10+ divergences and two tasks died in queue (repaired by operator cap raises + migration 0012's park/resume accounting fix). Child: Queue workers: give the non-code queues lane-style long-budget shifts (claim evaporation is a run-budget problem) #1130 — lane-style long-budget shifts for the non-code queues.
- Uncertified shift runs. Five runs on 2026-10-26 ended with a tool result lost, unable to certify their own completion even when the work landed; the parent re-verified from GitHub/queue state by hand each time. Child: Fleet observability: certify shift completion from queue state, not run self-report (uncertified runs) #1131 — certify completion from durable state, automatically.
Out of scope: producer jobs (new-issue-creator, pr-diff-review) and the Phase 3 sweep-producer expansion (paused by Drew 2026-10-26; separate decision).
Execution mode
Online: cron job bodies, worker instructions, monitor duties, Supabase migrations. All changes follow the fleet's isolation rules (pilot functions only; global claim/fail untouched) and the no-uncommitted-code rule.
Dependencies
Builds on migration 0012 (applied 2026-10-26: park/resume no longer consumes attempts). Measured by the queue-pilot interim check (2026-10-28) and the fleet monitor's hourly reports.
Acceptance
- Zero dead-in-queue tasks across all four queues over a rolling 7 days.
- Every shift delivery is certified from queue/GitHub state; no delivery rests on a run's self-report alone.
- The monitor's evaporation-repair count trends to zero; operator cap raises are no longer a routine event.
Filed by Muse (operator) 2026-10-26, grouping #1130 + #1131 under Drew's direction that fleet work carry epics, roadmap items, and dashboard prioritization.
Activity
developing-today commentedon Oct 27, 2026
CODE-365
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7219 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 3, 2026
Queue task 7219 picked up for fresh triage of the fleet-reliability epic. Revalidating its two named children (#1130 and #1131), current queue-health evidence, and whether the ongoing acceptance metrics can be certified today.
dezren39 commentedon Nov 3, 2026
Triage verdict — implementation landed; operating acceptance still needs a fresh measurement
Fresh GitHub state: this epic remains OPEN. Both named implementation children are CLOSED: #1130 (non-code lane shifts) and #1131 (state-based shift certification). #1130's Nov 1 closure note records the shifts live and the original tasks 1149/1157 terminal, but explicitly leaves the rolling zero-evaporation criteria to monitor verification. Its Oct 30 measurement had criteria 2 and 4 NOT MET and identified a full-day rerun follow-up task 4047 for Nov 3. #1131's Nov 1 closure note records the certifier, fallback summary, and 24h metric implemented; a six-lane dry run was all certified.
Current queue health (Nov 3 22:12Z) shows terminal-event counts over 7 days, but it does not report dead-in-queue/evaporation repairs or certify every shift in that window. Therefore the epic's numeric 7-day acceptance is not yet evidenced here, and I am not closing it or claiming zero evaporations. The implementation path has landed; the remaining work is the already-described monitor/full-day verification (task 4047 was cited by #1130; no duplicate task was created).
Disposition: #1233 stays OPEN for the operational measurement. No source changes or new PRs. Reconcile the full-day/following-week monitor data against the acceptance before closing this epic.