Skip to content

impl(pm-stack): quota watchdog escalation — fleet queue task + deduplicated incident issue for a browser-enabled agent #6033

Open

Description

@dezren39

Implements rung 4 of the browser-dependent quota watchdog (design #6031; core in the sibling watchdog-core issue). Separable from the core: the core detects and repairs; this issue is how a failure the ladder cannot fix becomes work for a browser-enabled agent — and, when only Drew can unblock it, becomes visible to Drew without delay.

Mechanism (per the design)

On escalation for a source, deduplicated by (source, failure class) — one open incident per pair:

  1. Fleet queue task enqueued (agent.tasks via fleet queue) with the full context packet, in the fleet's standing packet shape:
    • outcome wanted: "source <id> produces a fresh latest.json with real quota windows again"
    • failure context: class + class history, consecutive failures, the watchdog's persisted repairLog (every rung tried, with outcomes), producer history.jsonl tail reference, state-file path, adapter file reference
    • required capability, named so the claiming lane can actually do it: browser-enabled agent; laptop Helium via CDP (:19222); adapter origins only
    • acceptance check: a fresh producer document lands and the watchdog auto-closes the incident (the fixer does not close anything by hand)
    • stop conditions: a human-only challenge (2FA / verify-it's-you / passkey) → stop, mark human-blocked, do not work around it; anything outside the adapter's origin allowlist → stop and report on the incident issue
  2. GitHub incident issue opened in the repo that owns the adapter (Mistral: developing-today-forks/openchamber), cross-linked to design(pm-stack): browser-dependent quota watchdog — CDP scrape producers that self-repair, escalate to a browser-enabled agent #6031, labeled as a watchdog incident. This is the durable record and the fixing agent's reporting surface (FILE THE ISSUE discipline) — the queue task is the trigger, the issue is the record.
  3. Auto-close on recovery: the watchdog watches for the next producer success; on success it comments the resolution on the incident issue, closes it, and marks the queue task done. No human bookkeeping for recovery.
  4. Human-blocked surfacing: incidents whose blocker is a human-only step carry the flag/label the orchestration pass and stewardship sweep already use for NEEDS DREW items, and they surface immediately (not at the 24 h threshold) — a 2FA prompt is a 30-second Drew unblock, and the 2026-10-25 lesson (a sign-in wait sat unreported ~30 min) is the failure this prevents. Non-human escalations surface to Drew only if still unresolved 24 h after escalation (design open question 1 in design(pm-stack): browser-dependent quota watchdog — CDP scrape producers that self-repair, escalate to a browser-enabled agent #6031; threshold lives in one named constant).

Tradeoffs recorded in the design (why not the alternatives)

  • Issue-only escalation: durable but waits for a sweep — too slow as the sole trigger.
  • A dedicated cron spawning a browser agent on a flag file: guaranteed attention, but a standing job burning quota to watch an almost-always-empty inbox — rejected as primary (efficiency rule); revisit only if queue claim latency for browser-class tasks proves bad in practice, and say so on this issue if that evidence appears.

Acceptance criteria

  • Escalating a fixture incident creates exactly one queue task and one incident issue; a second escalation for the same (source, class) while the incident is open creates neither (dedup demonstrated).
  • The queue task packet contains the full context listed above — a browser-enabled agent can start from the packet alone.
  • A subsequent producer success auto-comments + auto-closes the incident issue and completes the queue task, with the watchdog's state incident cleared.
  • A human-blocked fixture is flagged for NEEDS DREW surfacing in the same tick it is marked, with the reason stated in one line.
  • An escalation unresolved at the threshold (24 h default) earns its NEEDS DREW line via the normal sweep path; before the threshold, Drew hears nothing (quiet-while-repairing rule).

Related: #6031 (design), #1260 (escalation split — Drew vs NEEDS ORCHESTRATOR), #2155 (orchestration residue in stewardship).

Activity

  1. added
    kind/featureNew feature or capability request
    app/pm-stackpm-stack (Pad/Grist) in developing-today/code
    on Nov 5, 2026
  2. developing-today commented on Nov 5, 2026

    @developing-today
    Contributor
  3. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    Addendum (design doc updated): for the NEEDS DREW surfacing mechanics — non-human escalations reach the orchestrator through the aggregator's existing #1148 watchdog tier items (watchdog:<input>, derived from input diagnoses) once the core (#6032) exposes the source state as a diagnosis; human-blocked promotion to Drew happens via the orchestrator's review of that item. The 24 h threshold and the queue-task + incident-issue mechanism are unchanged.

  4. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    on Nov 5, 2026
  5. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done

    Triage — task 10399, issue #6033

    Verified fresh: issue #6033 OPEN, labels include kind/feature, app/pm-stack, status/needs-triage — no hard-stop labels. Design issue #6031 is OPEN, so the design this issue says it implements ("per the design", rung 4) is not yet a closed/recorded decision; its open question 1 (24h surfacing threshold) is explicitly referenced by issue #6033 as still in the design.

    Routing: implementation is well-scoped once the design + the sibling watchdog-core land — acceptance is fixture-testable (dedup by (source, failure class) creating exactly one task + one incident issue, packet completeness, auto-close on producer success, human-blocked immediate surfacing, 24h threshold in one named constant). Sequence: issue #6031 design recorded → watchdog-core sibling → this escalation rung. Do not implement ahead of the core: this rung consumes the core's repairLog / incident state, which do not exist yet. No small_fix enqueued; issue #6033 stays OPEN, ready to route as an implementation task after the core.

    Outcome for task 10399: triaged.

  6. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/pm-stackpm-stack (Pad/Grist) in developing-today/codekind/featureNew feature or capability request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions