Repository navigation
Fleet observability: certify shift completion from queue state, not run self-report (uncertified runs) #1131
Description
Problem
Fleet shift runs keep ending "uncertified": a tool call or tool result inside the run never returns a success/failure status, so the run's own final report cannot confirm completion — even when every task it touched reached a verified terminal state. On 2026-10-26 this happened at least four times:
- 13:24 code worker shift: ended uncertified; outcomes (PRs refactor(id-web): drop dead leftmostGroup/leftmostCursorLine declarations (#952) #995, fix(id-web): preserve other marks on inline code on save (#953) #999 merged) verified afterwards from GitHub.
- Lane B 16:44 shift: 8 tasks handled, 2 PRs merged (verified), run outcome marked failed over one unconfirmed tool result.
- Lane B 17:24 shift: 6 tasks worked, completion uncertain per its own report; verified afterwards from queue health (39 done, 0 failed) and PR states.
- Lane A 18:24 shift: reported its summary as "already delivered" — it had never reached the chat; the parent had to re-deliver from live state.
Today the recovery pattern is manual: the parent re-verifies merges, closures, and queue counts from GitHub/Supabase after the fact and reports what actually happened. That works, but it costs a verification pass on every affected shift and leaves a window where nobody knows whether work landed.
Proposal
Certify shift completion from durable state, not from the run's self-report:
- A shift's completion record is the queue itself: per-task terminal states (complete/park/block with timestamps), the signal refresh, and the lane-lock release. A lightweight post-shift checker (the fleet monitor already reconciles queue ↔ GitHub hourly) compares the shift's claimed task set against terminal states and produces the certified summary line.
- When a run ends uncertified, the delivery automatically falls back to the state-derived summary ("uncertified run — outcomes verified from queue/GitHub state: …") instead of a bare failure, so the report Drew reads is always grounded.
- Track the uncertified-run rate per job as a fleet health metric; if it exceeds a threshold (say 2 in 24h for one job), the monitor flags it as an infrastructure problem in its summary rather than per-run noise.
Acceptance criteria
- A shift that ends with an unconfirmed tool result still produces a delivery whose every claim is derived from queue/GitHub state (no parent improvisation needed).
- The fleet monitor's summary includes, per shift job, runs completed vs uncertified for the last 24h.
- No shift outcome is reported as simply "failed" when its tasks verifiably reached terminal states.
Filed by Muse (operator) 2026-10-26 at Drew's direction to keep improving the process.
Activity
developing-today commentedon Oct 27, 2026
CODE-269
dezren39 commentedon Oct 27, 2026
More instances, same evening (2026-10-26, after this issue was filed):
That's six uncertified or undelivered shift reports in one day. In every case the work was verifiable from durable state (queue, sweep.log, GitHub) — which is precisely the certification source this issue proposes making automatic.
2 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
hidden_files/tools/shift-certifier.py(goal keep-the-four-repos-healthy) compares each lane's §8 handoff snapshot against live queue task states (batch PostgREST read via repo-queue.py) plus PR states viagh, and emits one CERTIFIED/UNCERTIFIED line per lane. Dry-run 2026-11-01T14:20Z against the six current lane snapshots: all six CERTIFIED (lanes a–f), including progressed-since-snapshot annotations (e.g. lane e task 1829 since re-claimed; lane d in-flight 4021/5591 since done).uncertified run — outcomes verified from queue/GitHub state: …with per-task queue/PR states — no parent improvisation needed; fleet monitor Duty 6 (added today via cron.update torepo-fleet-monitor, verified in the job definition) instructs the monitor to use that text verbatim and never report a shift as simply "failed" when its tasks reached terminal states.hidden_files/shift-certifications.jsonland prints per-shift-job completed vs uncertified for the last 24h (dry-run: shift-a..f each completed=1 uncertified=0); >2 uncertified/24h for one job raises an INFRA FLAG line, and Duty 6 requires it in the monitor summary.--summary-onlymode verified; GH-enabled run on lane f verified PR states joined (PR 3696/3712 MERGED, others OPEN as recorded).