Repository navigation
queue: claim_repo_task rejects lane openchamber-a (unknown lane_id) — OpenChamber hook worker cannot claim #6853
Description
Problem
The repo-health OpenChamber queue worker cannot claim any task: the queue lane registry rejects its registered lane id.
Verified 2026-11-07 ~08:07Z by the event-hook worker (repo-worker-openchamber, generation 1912):
fleet queue health(laptop binary) succeeds and reports the Turso store (fleet-queue-dezren39.aws-us-east-1.turso.io):repo-health-openchamber-pilot= 32 queued, 0 claimed, 5 parked, 11 blocked.fleet queue list --queue repo-health-openchamber-pilotsucceeds.fleet queue claim --queue repo-health-openchamber-pilot --worker repo-worker-openchamber --lane openchamber-a --lease-seconds 2700returns:claim_repo_task: unknown lane_id openchamber-a(broker exit 4).
openchamber-a is the lane id pinned in the worker runbook (same registered id the scheduled Lane A shift uses; the bare A form was rejected in the 2026-11-03 incident, and #4780 covered scheduled-lane UUIDs being rejected — this is the named lane on the current store).
Impact
Per the runbook §1/§0b, a lane-id rejection from a healthy backend is a verdict, not a backend failure: the worker stops and reports rather than retrying or switching to GitHub fallback. All 32 queued OpenChamber tasks are unclaimable by this worker until the registry is fixed.
Fix needed
Register/restore lane openchamber-a (worker repo-worker-openchamber, queue repo-health-openchamber-pilot) in the lane registry backing the Turso queue store, or publish the replacement lane id and update repo-queue-worker-instruction-openchamber.md to match. Verify with a live claim by the hook worker.
Acceptance criteria
-
fleet queue claim --queue repo-health-openchamber-pilot --worker repo-worker-openchamber --lane openchamber-a(and the code/opencode/nix lane equivalents) succeeds against the Turso store, or the lane registry is corrected so the pinned lane ids in the runbooks are the registered ones - A claim rejected for lane reasons is distinguishable in logs from a backend outage (verdict vs failure) without reading broker internals
- Regression coverage: a test or probe claims with each fleet lane id (code-a/b, opencode-a/b, openchamber-a/b, nix-a/b) and asserts no
unknown lane_id
Activity
developing-today commentedon Nov 7, 2026
CODE-2121
dezren39 commentedon Nov 7, 2026
Intake review
Related: this is the same rejection the scheduled Lane A/B shifts hit all morning 2026-11-07 (sweep log: nix-a, opencode-a, openchamber-a STOPs, all exit 4) and the binding blocker the fleet monitor recorded this run — Duty 1B bindings could not claim/park because the claim path is lane-based. Sibling history: issue #4780 (scheduled-lane UUIDs rejected). Risks: while lanes are unregistered, every queue lane is down and work proceeds only via GitHub fallback, which is how claim/label divergence accumulates. Suggested approach: dump the lane registry on the Turso store, diff it against the runbook-pinned ids, and either register the named lanes or re-pin the runbooks — one source of truth, verified by an actual claim per lane.
— Fleet monitor intake (repo-fleet-monitor, 2026-11-07 run): acceptance criteria appended to the issue body, labels normalized (priority/high, effort/m, status/needs-triage, meta/enriched), marked meta/enriched. No merged PR or commit was found this run that explicitly resolves this issue, so it stays open.
dezren39 commentedon Nov 7, 2026
Fixed 2026-11-07 (overnight lane 1, Drew approved)
Root cause: the Turso
fleet-queuestore was recreated fresh at the 2026-11-07 cutover and the task backup was loaded into it, but thelanesregistry table came over with schema only — zero rows. Every lane-validated claim therefore failed withclaim_repo_task: unknown lane_id <lane>(broker exit 4) on a healthy backend. There is no lane-registration verb in the fleet CLI or broker source (no INSERT path forlanesexists in pkgs/fleet), so the PG-era registry (agent.lanes, migration 0020_lane_identity) was the only historical copy, and it never migrated.Fix: registered all 32 fleet lanes directly in the Turso
lanestable (INSERT OR IGNORE, enabled=1). Worker/queue pairs were reconstructed from two independent in-store sources that agree exactly: the stamped lane_id/worker_id/queue history on claimed tasks in the same store, and the claim templates pinned in the four worker runbooks.Reversible: deleting those rows restores the prior state exactly; nothing else in the store was touched.
Proof (live claims, 120-second leases, all exit 0, lane_id stamped on the task):
The four proof leases expired at 23:49-23:50Z; per turso_queue.rs every claim runs expire_leases first, so the tasks return to their queues on the next shift claim and no task work was performed on them. The remaining lanes differ only by registry rows of identical shape, read back and verified present (32 rows: 26/2/2/2 per queue); the next scheduled shifts exercise them live, and any residual lane rejection would surface in the watchdog within one cycle.
Still open from the acceptance criteria: a durable regression probe (a test that claims with each fleet lane id) and the lane-vs-outage log distinction. Suggest filing the probe as its own small follow-up so this issue can close on the registry fix itself; the registry half of criterion 1 is met for all four lane families.
dezren39 commentedon Nov 8, 2026
Restore-completeness audit — the snapshot HAD the lanes; the export never carried them (2026-11-07 18:21 CST, Drew's correction)
Drew's point stands and is now proven at the source: the lane registry was supposed to be filled from the 2026-11-06 snapshot. Audit of that snapshot vs the live stores:
Snapshot of record: the fleet DIY backup of muse-dev2, created 2026-11-06T10:47:11Z (R2
pg/muse-dev2, sha2567bd52e20…), retained on the laptop at~/workspace/scratch/muse-dev2-queue-core-export-2026-11-06/(full pgdump + per-table export + manifest.json).Root cause, sharpened: the cutover's per-table export (manifest.json) contains exactly five tables — tasks, leases, workers, checkpoints, repo_task_lane_ledger.
agent.laneswas never exported, so the load had nothing to skip — the table was dropped at export time, not load time. The snapshot'sagent.lanesholds 45 rows: 30 named lanes + 15 UUID-keyed lanes (surfaceopenchamber, registered 2026-11-02). Tonight's fix had reconstructed 32 from in-store history + runbooks.fleet-queue diff (snapshot → live, post-audit)
agent_events_pg_historyLane reconciliation: the snapshot's 30 named lanes (code-a…z, nix-a, openchamber-a/b, opencode-a) were ALL already in tonight's 32, with identical worker/queue pairs. Tonight's set adds nix-b + opencode-b (from the runbooks; not in the Nov-6 registry — kept, additive). The snapshot's 15 UUID lanes were missing. Backfilled additively (INSERT OR IGNORE generated directly from the dump, all 45 snapshot rows applied): lanes 32 → 47, 47/47 enabled, UUID lanes 0 → 15. Post-backfill health: claims flowing (code pilot 157 claimed, done 4,977 and rising).
Second gap —
agent.token_expirations(10 rows) has NO Turso home. The credential-expiry tracker (created 2026-11-03, migration 0023; names/services/expiry dates only, never values) was in the audit's agent queue-core class, but no table exists for it in fleet-queue or fleet-platform. Its 10 rows survive only in the snapshot. Not backfilled here: choosing its store is a design call, so it is dispositioned in the restore-procedure issue filed with this audit (recommendation recorded there).Sibling stores verified (same snapshot, other destinations): fleet-platform
agent_external_comms= 69, exact (Lane C top-up) ·agent_quota_records= 11,904 ≥ snapshot 10,513 ✓.Spot-check: the morning's other recreates — CLEAN
turso db export → import(the CLI has no rename, turso CLI: no rename and no libsql->TursoDB mode conversion — every fix is a full export/import data move #6827), which carries every table by construction, and their flip counts were verified (ci_events parity 11,199 = 11,199; mirror 225 / 22 / 33,842 exact). Only fleet-queue's hand-picked per-table export could silently drop a table — and did. Observation, not a gap: Turso's mirror sets carry the exporter's designed 21–22 tables (export_state = 22 rows in both stores) vs PG's 28mirror_*tables, which include PG-side history/definitions tables that were never exporter output.Procedure fix filed: the restore-procedure issue linked below (reference-table checklist + post-restore row-count diff gate + token_expirations disposition).