fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369

Activity

1. dezren39

   added

   priority/highImportant and time-sensitiveImportant and time-sensitive

   area/infraInfrastructure, deployment, and operationsInfrastructure, deployment, and operations

   kind/infrastructureInfrastructure / host-portability workInfrastructure / host-portability work

   on Nov 5, 2026
2. developing-today commented on Nov 5, 2026

   

   developing-todayBot

   on Nov 5, 2026 – with developing.today

   Contributor

   More actions

   CODE-1887
3. dezren39

   added

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   task/triageQueue packet kind: verify, reproduce, and route this itemQueue packet kind: verify, reproduce, and route this item

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   and removed

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Nov 6, 2026
4. dezren39 commented on Nov 6, 2026

   

   dezren39

   on Nov 6, 2026

   MemberAuthor

   More actions

   🤖 Agent note · picked up

   • Agent: queue worker (repo-worker-code), lane code-n
   • Session / ID: task 11087
   • Found via: queue claim
   • Task: Triaging issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 — verify at current main whether broker-transport outage alerting already exists, scope the fix path, and record blockers/ordering.
   • Expectation: s — triage done when findings are posted with evidence and a routed follow-up (or closure evidence if already handled).
   • Links: task 11087 (queue repo-health-code-pilot)
5. dezren39 commented on Nov 6, 2026

   

   dezren39

   on Nov 6, 2026

   MemberAuthor

   More actions

   Triage — task 11087 (lane code-n, 2026-11-06)

   Verdict: CONFIRMED — real, not already handled, not superseded. Fix path scoped below; follow-up fix task enqueued from this triage.

   Evidence (fresh, this run)

   • Issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 is OPEN (verified 2026-11-06T13:28Z via gh), labels task/triage, priority/high, area/infra, kind/infrastructure; no hard-stop labels (held, user/owned, do-not-merge, do-not-auto-edit).
   • Transport is healthy right now: this shift's Step 0 fleet queue health --queue repo-health-code-pilot succeeded at 2026-11-06T13:27Z (queued 242 / claimed 76 / parked 80) and task 11087 claimed normally — so this is a detection gap, not a live outage.
   • Origin incident issue incident: repo-queue broker tunnel down 2026-11-04 23:11 UTC — worker lanes cannot claim (session proxy CONNECT closed) #5708 is CLOSED — it was the hand-filed incident itself; nothing in it implements automated detection.
   • Search of open issues for broker-transport alerting finds only issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 itself (plus unrelated broker hardening issue fleet brokers: review-finding hardening for dual-bind transports (from PR #6514) #6515 and the broker design issue design: repo-queue + secrets brokers on the laptop behind the session proxy #5362) — no duplicate and no in-flight implementation.

   Why the gap still exists (current mechanisms checked)

   • fleet-watchdog (every 15m, built for issue Babysitter coverage gap: the cron fleet's own agent sessions have no real-time watcher #1358) watches cron job runs via live scheduler state and alerts on stuck/failing/silent jobs. It never exercises fleet queue health, so a transport outage whose symptom is lanes exiting silently at the Step 0 gate is not directly probed, no deduped transport-incident issue is filed, and there is no transport-specific recovery signal.
   • repo-fleet-monitor (every 2h) does call fleet queue health, but only inside duties that themselves depend on the queue — if the transport is down the monitor cannot do that work — and a 2h cadence cannot meet the ~15-minute alert acceptance criterion anyway.

   Scoped fix (recommended path)

   Add a transport probe duty to the existing fleet-watchdog (no new job — fewer jobs that do more):

   1. Each watchdog run executes fleet queue health --queue repo-health-code-pilot end-to-end from the sandbox and records the result in its state/log.
   2. On failure, re-probe once after ~90 seconds within the same run. Two consecutive in-run failures = outage (this filters the ~16s self-healing restart window noted in the secondary observation on issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369, while still alerting inside one 15m watchdog cycle).
   3. On outage: alert in the main chat AND file ONE incident issue via gh (the reporting path — chat + gh — does not depend on the queue), deduped by searching open issues for an existing incident: repo-queue broker issue for the same outage window first (the incident: repo-queue broker tunnel down 2026-11-04 23:11 UTC — worker lanes cannot claim (session proxy CONNECT closed) #5708 pattern).
   4. On the first healthy probe after an alerted outage: post ONE recovery line, so a silent gap is distinguishable from a healthy idle queue.
   5. Mechanics: watchdog body changes follow the house rule — edit the repo copy first, commit, then apply the identical text via cron.update. No cadence/rate change is needed or proposed.

   Ordering / blockers

   • No Drew decision needed: the ask, host (watchdog), and alert shape are all specified on issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369.
   • Effort: s–m (one job-body duty + a small state field for outage-window dedupe).
   • Acceptance mapping: in-run double-probe ⇒ alert ≤ ~15m ✓; dedupe search before filing ⇒ no duplicate incident issues ✓; probe reports via chat/gh, never via the queue ⇒ check does not depend on the queue it monitors ✓.

   🤖 Agent note · done (triage)

   • Agent: queue worker (repo-worker-code), lane code-n
   • Session / ID: task 11087
   • Found via: queue claim
   • Task: Triaged issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 — confirmed the broker-transport alerting gap at current state, scoped the watchdog-probe fix, enqueued the follow-up fix task.
   • Expectation: triage complete; implementation continues in the follow-up fix task for issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369.
   • Links: task 11087 (triage, this run); follow-up fix task enqueued with dedupe key repo-health:v1:fix:developing-today/code:issue:6369
6. dezren39

   added

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   and removed

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Nov 6, 2026
7. dezren39 commented on Nov 6, 2026

   

   dezren39

   on Nov 6, 2026

   MemberAuthor

   More actions

   🤖 Agent note · handed off

   • Agent: queue worker (repo-worker-code), lane code-a
   • Session / ID: task 11161
   • Found via: queue claim
   • Task: Handing off task 11161 for issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 to the opencode execution lane per instruction §1b; verified issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 is OPEN with no held, user/owned, do-not-merge, or do-not-auto-edit labels.
   • Expectation: opencode-claimant launches the bound opencode session to drive task 11161 to a terminal state.
   • Links: issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369
8. dezren39

   added

   queue/parkedQueue task is parked (waiting on CI, time, or an event)Queue task is parked (waiting on CI, time, or an event)

   status/parkedDeliberately paused with a recorded resume conditionDeliberately paused with a recorded resume condition

   and removed

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   on Nov 6, 2026
9. dezren39 commented on Nov 6, 2026

   

   dezren39

   on Nov 6, 2026

   MemberAuthor

   More actions

   🤖 Agent note · handed off

   • Agent: queue worker (repo-worker-code), lane code-a
   • Session / ID: task 11161
   • Found via: queue claim
   • Task: handing task 11161 (issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369) to the opencode claimant per execution-lane assignment; claimant launches the bound opencode session
   • Expectation: claimant picks up task 11161; if not launched before lease/park expiry, task 11161 returns to the queue
   • Links: issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369
10. dezren39 commented on Nov 6, 2026

    

    dezren39

    on Nov 6, 2026

    MemberAuthor

    More actions

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-f
    • Session / ID: task 11161
    • Found via: queue claim
    • Task: task 11161 (small_fix, issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369) — execution-lane assignment returned opencode; handing off to the opencode-claimant per worker instruction §1b, parked pending claimant launch. Issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369 re-verified OPEN with no hard-stop labels via gh this shift. Attempts now 4/6.
    • Expectation: claimant launches the bound opencode session and drives task 11161 to a terminal state
    • Links: issue fleet(queue): alert when the repo-queue broker transport is down — lanes currently starve silently at the Step 0 gate #6369