Repository navigation
Babysitter coverage gap: the cron fleet's own agent sessions have no real-time watcher #1358
Description
The gap (found in fleet review, 2026-10-27)
The babysitter (openchamber-babysitter, every 20m) watches OpenCode/OpenChamber sessions: verdicts, stall detection, nudges. But the fleet's other working sessions — the cron agent runs themselves (lane shifts, producers, monitor, steward, dispatcher) — have no real-time watcher at all.
What covers them today: the fleet monitor's hourly reconciliation, which finds claim evaporation after the fact (labels repaired, handoffs posted), and the lane gate/lock discipline. On 2026-10-26 that meant evaporation on the opencode/openchamber queues was discovered up to an hour late — task 1149 (#530) burned multiple attempts before its final one died at 7/7 — and several shift reports never arrived in chat at all (now tracked as the certification problem, #1131). Detection latency, not detection absence, is the gap.
Scope (assess, then build or formally accept)
- Assess a lightweight run-watch for cron executions: a run sitting in its execute phase far beyond its job family's p90 duration (baselines exist in run history) gets flagged within ~20 minutes — e.g. a small periodic check over scheduler run state, or a heartbeat convention lane runs touch at checkpoint time that a watcher can staleness-test.
- The watcher's output should be the same shape as the monitor's repairs (flag + evidence), just faster; intervention stays with the existing owners (monitor repairs, lanes re-claim).
- Alternatively, if the assessment shows the hourly bound is acceptable — e.g. because migration 0012 + lane discipline made evaporation cheap and rare — record that reasoning here with the evaporation census numbers that support it, and close this as a documented acceptance.
Acceptance criteria
- An assessment exists (comment on this issue) with run-duration baselines per job family and the observed detection latencies for the 2026-10-26 evaporation cases.
- Either a run-watch mechanism is built and demonstrated flagging a stale run inside ~20 min, or the hourly bound is explicitly accepted with the supporting numbers recorded.
- The decision is reflected in the babysitter/monitor instruction files (who watches what, at what latency).
Filed by Muse (operator) 2026-10-27, from the cadence/scope review with Drew.
Activity
developing-today commentedon Oct 27, 2026
CODE-458
dezren39 commentedon Oct 27, 2026
Mechanism built (2026-10-27, Drew's 'improve #2'):
fleet-watchdogjob, every 15 min — checks every fleet job's scheduler state for stuck runs (per-family age thresholds), failure streaks (≥2), silent jobs (no run in 2× cadence), and watcher gaps (any babysitter or fleet-monitor failure, or a >2-cycle gap, is highest priority). Alert-only by design: it never restarts or edits a job (auto-restart can duplicate side effects); alerts land in the fleet/token chat and the watchdog log. This issue stays open for its assessment half: whether the 15-minute alert bound + hourly monitor reconciliation is sufficient, or the fleet needs run-phase heartbeats pushed from the jobs themselves.75 remaining items
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: parked
task: 6496 (repo-health-code-pilot; worker repo-worker-code)
state: parked until 2026-11-04T08:37Z (30 minutes)
lane: bf419306-c4ef-470f-a553-beabf93239de
verified: #1358 remains OPEN with no open PR or hard-stop label; #2132 remains OPEN and its 48-hour comparison is still assigned to task 8468 per its latest note; #2153 remains OPEN with the conductor cutover reported complete. The laptop has cached fleet-watchdog/conductor/babysitter logs through 2026-11-04 01:26 CST, but they contain sampled summaries, not per-family run-history rows or the exact 2026-10-26 evaporation/alert timestamps. No fleet scheduler timer or local muse.db/run-history source is present, and repo-queue.py has no run-history verb. Also, #530 is an unrelated merged UTS-46 PR, so the packet's task 1149 / #530 reference needs its repository URL to disambiguate.
blocker: Cannot produce defensible p90 baselines, detection latencies, or the accept-or-extend decision without the scheduler history and timestamped events; no instruction files were edited and no numbers were invented.
resume: Provide a credential-free scheduler-history export with per-family run durations and timestamps for each 2026-10-26 run start, evaporation, monitor discovery, and watchdog alert; include task 1149's repository URL. Resume after task 8468 posts or dispositions the #2132 cost comparison, then decide on #1358 and update the watcher instructions.
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
opencode, so this task is handed off — the laptop claimant launches the bound opencode session, which drives task 6496 to its terminal state. Fresh check this shift: issue Babysitter coverage gap: the cron fleet's own agent sessions have no real-time watcher #1358 is OPEN with no hard-stop labels (no held / user/owned / do-not-merge / do-not-auto-edit). Prior progress preserved: baselines/detection-latency evidence still missing; scheduler-history export is the resume condition.dezren39 commentedon Nov 4, 2026
agent-note:v1
{"kind":"handoff","task":6496,"lane":"code-v","execution_lane":"opencode","state":"parked","note":"task 6496 handed off to opencode claimant by lane code-v. Fresh revalidation: issue #1358 OPEN, no hard-stop labels, no open PR for this issue per prior progress. Prior park evidence preserved: scheduler run-history source for baselines/detection latencies was not found locally or on the laptop — claimant resumes the assessment from task progress."}