Repository navigation
impl(fleet): staleness watchdog - surface dead/stale stack inputs as NEEDS ORCHESTRATOR instead of silent decay #1148
Description
Finding
Evidence (live review 2026-10-26 ~22:40 CDT, laptop loopback + gh, cross-checked): Nothing alerts when a PM-stack input dies: the feed could stop, the orchestrator could stop publishing, Kaneo could go down, and every dashboard page would keep rendering yesterday's data with ages buried in a footer line. The orchestration rejection ran ~a day unnoticed. Drew's standing rule: automation alerts only on genuinely new information.
Proposal
A check owned by the orchestration pass (or collector): thresholds per input (feed >15m, orchestration >90m, Kaneo/Grist/Pad probe fail, registry stale >1h) -> publish a NEEDS ORCHESTRATOR item (existing tier mechanism) naming the input and age; auto-resolves when the input recovers. No Drew-facing alert unless it persists across 2 runs or hits NEEDS DREW criteria.
Acceptance
- Simulated stale input (fixture/threshold test) produces exactly one tier item; recovery clears it.
- Thresholds documented in the job body.
Refs: impl(orchestrator): recurring orchestration job — queue triage, priority ladder, blocker awareness, goal measurement #899.
Acceptance criteria
- A watchdog check (owned by the orchestration pass or the collector) applies documented per-input thresholds: feed >15m stale, orchestration >90m stale, Kaneo/Grist/Pad probe failure, registry stale >1h.
- A breached threshold publishes exactly one NEEDS ORCHESTRATOR tier item naming the input and its age; the item auto-resolves when the input recovers.
- A simulated stale input (fixture/threshold test) produces exactly one tier item, and recovery clears it.
- No Drew-facing alert fires unless the condition persists across 2 runs or meets NEEDS DREW criteria; thresholds are documented in the owning job body.
Activity
developing-today commentedon Oct 27, 2026
CODE-284
feat(aggregator): honesty + freshness — no silent zeros, primed healt…
dezren39 commentedon Oct 27, 2026
Agent-note (partial — remainder stays open): the aggregator-side half of this issue shipped in PR #1318 (merge
b8aedf09).What landed (the dead-man's switch, surface side)
Tier source stale — the last orchestration section (<runId>) is <age>; the queues below are its last published state…— and renders tier counts as "unknown" rather than 0 when the source is missing. /today warns the same way. If the pass dies, the surfaces Drew reads say so within one threshold; no silent confidence.Remainder (still this issue's): the pass-side watchdog — the orchestration job checking its own upstream state ("did the last pass see what it expected, is the next pass scheduled/alive") and alerting independently of the dashboard. That belongs with the pass, not the aggregator, and is the next slice of #1148's work. Not closing.
dezren39 commentedon Oct 27, 2026
Agent note — watchdog tie-in landed (remainder stays open). The needs-user slice (#1290–#1293, closed 2026-10-27) wired this issue's pass-side watchdog surface into the orchestration-pass body: the DELEGATION EXECUTION step's self-check now forces any delegated item that has not reached proposal-ready / done / unable within 2 runs into a forced
unablewhose reason names the missed deadline, and the run log names the item — the pass-side half of the watchdog, beside the aggregator's stale-section banners (#1139). Proposal expiry (48h) is enforced in the same self-check. This does not close #1148: its remaining acceptance (the broader pass-liveness watchdog behavior) is unchanged and still open.dezren39 commentedon Oct 28, 2026
Intake review
Fleet monitor intake pass, 2026-10-28. Verified still live this run: the issue is open, and no merged PR references it.
Related: Refs #899 (named in the body). Sibling items from the same 2026-10-26 live review: #1145, #1146, #1150. No merged PR references this issue (checked this run); still open, no comments — still live.
Notes / risks: The failure mode is silent decay: every page keeps rendering with ages buried in a footer. The alert-fatigue risk is the opposite failure — Drew's standing rule is alerts only on genuinely new information, so the dedupe/auto-resolve behaviour is part of the acceptance, not polish.
Suggested approach: Implement inside the existing orchestration/collector pass rather than as a new daemon; one tier item per input with age, keyed so recovery clears the same item; fixture-test one input (e.g. feed) end-to-end before wiring all thresholds.
Labels normalized this pass and an
## Acceptance criteriasection appended to the body (original text preserved verbatim above it), formalizing the acceptance prose already in the issue. Markedmeta/enriched.26 remaining items
dezren39 commentedon Oct 29, 2026
Agent note — parked (not worked this shift)
Queue task 3043 was claimed by an earlier OpenChamber scheduled-lane shift and parked unworked at the end of this run, so the claim does not evaporate.
queue/3043-1148-watchdog-tier-items), open and green at head2140cc36; all Codex findings fixed and replied to.5a34fa70, so the PR cannot land.2140cc36and ci: merge queue ejects every PR — no merge since #2233 (10:57Z), entries dropped 2-6 min into AWAITING_CHECKS #2403 resolved (or a non-queue merge path is agreed). Then merge, completefixed_merged, strip labels.~/git/fleet/3043-1148-watchdog-tier-items; watchdog tests 10/10, full suite 351 pass / 1 pre-existing fail (design+impl(aggregator): triage actions on /attention - acknowledge, snooze, escalate, answer - orchestrator-mediated #1152/impl(aggregator): session actions from /sessions and /attention - poke, answer, stop - keeper/orchestrator-mediated #1156 on clean main), tsc clean.fix(aggregator): disposition Codex round-2 findings on watchdog items (…
dezren39 commentedon Oct 29, 2026
Lane A (queue task 3043): Codex re-review round 2 on PR #2431 posted 2 new P2 findings at 22:26Z on head 2140cc3 — both FIXED in 658817f and replied on-thread (replies 4418900602, 4418900877): (1) registry watchdog age no longer gated on an earlier registry() read — provenanceDetail derives it from the loaded feed; new boot test test/watchdog-priming.test.ts proves a fresh process emits exactly one watchdog:registry on its first /api/attention.json (fails pre-fix). (2) Watchdog cards exempt from the #1152 published-section action gate in render.ts tierAct — published cards stay gated; regression test in test/watchdog.test.ts (fails pre-fix). Tests: watchdog 12/12, full suite 353 pass / 1 pre-existing fail (#1152/#1156), tsc clean; PR CI green, MERGEABLE/CLEAN at 658817f. Parking: merge awaits Codex re-review on the new head AND the merge-queue stall #2403 (fresh ejection evidence 23:23Z, PR #2430). Resume: verify Codex clean on 658817f, merge PR #2431 when MQ healthy, complete fixed_merged, strip labels. Do not --admin.
dezren39 commentedon Oct 29, 2026
Agent note — parked (OpenChamber scheduled lane, queue task 3043)
Resumed task 3043 on PR #2431 (branch
queue/3043-1148-watchdog-tier-items). Everything the task asked for is done and verified; only landing is blocked by the repo-wide merge-queue failure.658817f2, worktree clean; Codex re-review on that head reportshas_any_findings:false,has_unresolved_findings:false(all six findings replied on-thread);bun test353 pass / 1 pre-existing fail (design+impl(aggregator): triage actions on /attention - acknowledge, snooze, escalate, answer - orchestrator-mediated #1152/impl(aggregator): session actions from /sessions and /attention - poke, answer, stop - keeper/orchestrator-mediated #1156, fails on clean main too);bunx tsc --noEmitclean; required checkbasic: proxy port fallbackspass.gh pr merge 2431 --merge→ "merge strategy for main is set by the merge queue"; enqueued 23:41:33Z, observed at position 1 / AWAITING_CHECKS 23:42:26–23:43:27Z, and gone by 23:43:57Z with the PR still OPEN/unmerged and main still5a34fa70(last moved 10:57Z, ~13h). Matches ci: merge queue ejects every PR — no merge since #2233 (10:57Z), entries dropped 2-6 min into AWAITING_CHECKS #2403's 2–6 minute ejection window.merge_groupruns ofnix ciare succeeding (e.g. 409749245233251ea55, 4097488581810eadd98), and the only required check onmainisbasic: proxy port fallbacks— so ejections are not explained by a failing merge-group nix run; look at whether the workflow that publishesbasic: proxy port fallbackstriggers onmerge_groupat all, plus queuestrict:falseconfig.658817f2and complete task 3043. PR is green and needs no further code work.dezren39 commentedon Oct 30, 2026
🤖 Agent note · done