impl(pm-stack): browser-dependent quota watchdog core — disk state, failure classes, repair ladder #6032
Description
Implements the watchdog core from the design in #6031 (browser-dependent quota watchdog). Scope is the generic machinery only — no provider specifics; Mistral is adapter #1 on top of this (openchamber issue, linked from #6031).
What to build
In pm-stack (this repo), deployed on the laptop beside the gatherer:
- Watchdog oneshot + timer (
pm-stack/systemd/): runs on the gatherer's 15-minute rhythm, offset (e.g.*:3/15) so it judges fresh producer documents. Oneshot discipline like the existing laptop watchdog family (laptop-backups-watchdog code#1841, codex-remote-control-watchdog code#1845). - State file
~/.local/state/pm-aggregator/quota-watchdog-state.json— atomic tmp+rename writes; per-source health record:lastAttemptAt,lastSuccessAt,consecutiveFailures,currentClass, sanitizedlastDetail,cooldownUntil,repairLog(last 20),incidentrefs. Nothing decision-relevant lives in process memory — this is code#5928's lesson applied from day one (cross-tick cooldown + failure counts on disk). - Failure-class evaluation: read each registered source's
~/atlas/<provider>-export/latest.json+ the gatherer heartbeat; classify intosession-expired/cookie-missing/endpoint-changed/browser-down/cdp-unreachable/rate-limited/unknown(producer pre-classifies its own failures; the watchdog owns classes the producer cannot see, e.g. missing document, browser process state). - Repair ladder executor with per-rung attempt caps (max 2 per incident) and a persisted repair log, cheapest first: (0) rate-limit cooldown honored from disk; (1) retry next tick (no-op rung); (2) CDP session re-validation against the live laptop browser (cookie presence by name; in-page fetch re-run — the cookie value is not read into the process; extraction only for adapters that declare a header-token path, value → fleet secrets by registered name only); (3) browser re-login driver via the provider's own SSO flow using the browser's standing Google session — stops dead at a human-only challenge and marks the incident human-blocked. Rung 4 (escalation) is the sibling escalation issue, invoked from here.
- Adapter loader: read per-provider adapter files (
pm-stack/exporters/adapters/) — origins allowlist, CDP port, data-path order, cookie names, login hints, class map, thresholds. The core contains zero provider names. - Producer scaffolding: a forkable producer template from the retired Meta producer (
~/atlas/meta-quota-export/meta-quota-export.mjson the laptop — CDP plumbing, tab ownership per code#5121, per-call timeouts, failure-class exit codes, atomic latest.json + history.jsonl), landing producer source inpm-stack/exporters/; plus the gathererFILE_EXPORTSregistration step documented/wired for each adopted source. - Health history: append per-tick per-source observations to
~/.local/state/pm-aggregator/quota-watchdog-history.jsonl(interim); emit into the usage snapshot pipeline once code#4889's laptop path bug is fixed (do not block on it).
Safety bounds (from the design)
Never writes quota data; never changes credentials other than re-storing an extracted session under its registered secret name; browser automation only against adapter allowlist origins, read endpoints + login flow only; no secret value in any log, state field, or document; login attempt caps double as account-lockout protection.
Acceptance criteria
- Watchdog oneshot + timer installed on the laptop and ticking on the gatherer cadence; state file written atomically each tick and surviving across invocations (demonstrated over ≥3 consecutive ticks).
- A synthetic broken source (fixture adapter + producer stub) walks the ladder: retry → CDP re-validation → re-login attempt → escalation hand-off, with each rung's attempt recorded in
repairLogand caps honored (no rung attempted >2× per incident). - A
rate-limitedfixture setscooldownUntilon disk and the next tick honors it without any browser action (the quota: claude read needs cross-tick memory — usage-endpoint 429 (proven) + oneshot gatherer defeats the fetcher's in-process cooldown/cache #5928 regression test, in effect). - Human-only challenge fixture marks the incident human-blocked and the state carries the surfacing flag the orchestration/stewardship sweeps read.
- Quiet when healthy: a healthy source produces no output beyond journal/state/history records.
- Tests name their oracles per the testing doctrine (state-machine invariants for the ladder are a natural PBT/MBT fit).
Related: #6031 (design), #1260 (self-healing umbrella), #5928 (cross-tick state), #4889 (snapshot history path).
Activity
developing-today commentedon Nov 5, 2026
CODE-1697
dezren39 commentedon Nov 5, 2026
Addendum (design doc updated): the surfacing channel for a failed/escalated source is the aggregator's existing #1148 watchdog mechanism —
pm-stack/aggregator/src/watchdog.tsderives dedupedwatchdog:<input>tier items from input diagnoses, merged into the orchestrator's tiers byderive.ts. Read the human-blocked acceptance criterion as: the source's state is exposed as an aggregator input diagnosis so it surfaces through that channel (adding a quota-source diagnosis to the health layer is in scope if none exists), with the orchestrator's review promoting human-blocked items to Drew. Separately: fleet-watchdog (#1358) watches Muse scheduler jobs only (alert-only) — it is not this watchdog's supervisor and is not a dependency.dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
opencodebyfleet queue lanefor task 10398; the laptop opencode claimant launches the bound session, which drives task 10398 to its terminal state. This lane does not work it directly.dezren39 commentedon Nov 5, 2026
Task 10398 (lane code-s) claimed and assigned to the opencode execution lane. Handed off to the opencode claimant — the bound session drives this task to its terminal state. Parked by this lane per the handoff protocol; issue labels already carry the parked set.
25 remaining items
developing-today commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
Triage findings — current main
pm-stack/exporters/is the design's default home; the open question can be settled by the core implementer before code lands and does not block start.repairLogcontract before integrating the rung-4 handoff. openchamber#1630 remains blocked on the named Mistral sign-in/data-path result. Existing impl(fleet): staleness watchdog - surface dead/stale stack inputs as NEEDS ORCHESTRATOR instead of silent decay #1148 aggregator diagnosis/NEEDS DREW surfacing is the channel identified in the issue addendum; code#5928's disk-state lesson applies, and append-usage-snapshots.ts fails on laptop: quota-view.md path ENOENT under ~/workspace #4889's snapshot-path bug is explicitly non-blocking.dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
opencodedezren39 commentedon Nov 5, 2026
agent-note:v1
OpenChamber scheduled lane (lane ID: 9aca43b4-5439-4939-9f71-323f5d32521b) picked up this issue for triage at 2026-11-05T20:20:16Z.
Task 10398 was claimed and handed off to opencode claimant for direct resolution. The queue task is now in 'claimed' state with checkpoint documenting the handoff.
Labels updated: removed queue/parked and status/parked, added queue/claimed and status/in-progress.