Repository navigation
Queue workers: give the non-code queues lane-style long-budget shifts (claim evaporation is a run-budget problem) #1130
Description
Problem
The three non-code queues (repo-health-opencode-pilot, repo-health-openchamber-pilot, repo-health-nix-pilot) are worked by short-budget arrangements — a 2-hourly agent-queue-worker cron and per-queue hook workers — while the code queue runs two staggered lane shifts (40-min cadence, ~90-min budget, bundle of up to 10, per-lane locks, atomic claims). The short-budget workers keep dying mid-task: they claim, label, do real work, and their run ends without a terminal verb (complete/park/block). The lease expires, labels and queue state diverge, and the fleet monitor spends a duty repairing the divergence instead of the fleet making progress.
Evidence (2026-10-26 alone)
- Task 1149 (opencode lab-ops-hub: judge container crawl hosts on the full UTS-46 mapped form (#516, #526) #530): evaporated 4 times, went dead-in-queue at 4/4; after an operator cap raise it was claimed again and the run was cut at its resolve cutoff mid-typecheck — branch unpushed, task left claimed. Its work has been verified-complete at checkpoint level (
targeted_tests_passed) for most of a day without landing. - Task 1157 (openchamber fix(bin): validate ANTIGRAVITY_LS_ADDRESS port strictly (#494) #495): evaporated 4 times, dead-in-queue at 4/4 (cap raised 2026-10-26 evening).
- Task 1160 (openchamber lab-ops-hub: Worker-fallback crawler has no fetch timeout — a slow target hangs the crawl indefinitely #500): reached its 2/2 cap the same way.
- Task 1187 (nix ci: scheduled flake-inputs-update workflow (openchamber + opencode) #907) and task 1177 (opencode id(roc): F11.18 pin the Roc compiler for checked-in example modules and rebuild them in CI #541): further evaporations the monitor repaired the same day.
- The fleet monitor repaired 10+ claim/label divergences on 2026-10-26 across the non-code queues; the code queue, on lane shifts, had one in-shift evaporation (task 1216) which the same shift self-recovered.
Migration 0012 (2026-10-26) fixed the attempt accounting (park/resume no longer consumes attempts), but evaporation itself is a run-budget problem: a worker whose run can be cut at any resolve/timeout boundary will keep stranding claims no matter how the attempts are counted.
Proposal
Give each non-code queue the lane treatment the code queue proved out on 2026-10-26:
- One cron-driven shift worker per queue (or one worker rotating queues per shift), 40–60 min cadence, ~90-min run budget, bundle of up to 5 tasks strictly one at a time.
- Per-queue lock file; a lane gates out only on its own live lock.
- Same worker instruction as the code lanes (claim → checkpoint on claim → outcome-early ordering → park instead of waiting on CI → verified terminal state before claiming the next), including the §10 anti-evaporation discipline.
- Retire or repurpose the short-budget hook workers for those queues once the shifts are live (keep them as fallback, as the code hook was kept).
Acceptance criteria
- Shift worker(s) exist for the opencode, openchamber, and nix queues with the lane properties above (cadence, budget, bundle, own-lock gating).
- A full day of operation with zero claim evaporations on the non-code queues (monitor Duty 1 addendum finds no divergences to repair), or every evaporation traced to a named infrastructure cutoff with the task recovered by the next shift without operator help.
- Tasks 1149 and 1157 reach terminal states through the new workers without further operator cap raises.
- The fleet monitor's per-run evaporation-repair count for non-code queues trends to zero over the following week (measured at the queue-pilot interim check).
Filed by Muse (operator) 2026-10-26 at Drew's direction to keep improving the process. Related: migration 0012 (attempt accounting), the 2026-10-28 queue-pilot interim check.
Activity
developing-today commentedon Oct 27, 2026
CODE-268
dezren39 commentedon Oct 27, 2026
Update (2026-10-27, Drew's scale-out direction): the lane half of this issue is being built now as #1335 under epic #1334 — A/B shifts for the opencode, openchamber, and nix queues on the code-lane pattern. This issue stays open for its remaining scope (the fallback 2-hourly worker's retirement criteria and the zero-repairs/day acceptance) and as the design record.
16 remaining items
dezren39 commentedon Nov 1, 2026
agent-note:v1
task: 5529 (triage, lane A)
state: triaged — IMPLEMENTED; closing with evidence
checked_at: 2026-11-01T12:36Z
Acceptance check against live state:
Closing as implemented. The old 2-hourly agent-queue-worker cron remains enabled as a general pilot worker (fallback role, as the proposal allowed for the retired hook workers).
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done