Repository navigation
design(pm-stack): SLOs + self-healing for the PM stack itself — freshness SLOs with error budgets, watchdog remediates before alerting #1260
Description
Part of epic #1184 (PM stack coherence — ideal-system design series, 2026-10-26).
Why
The stack's own reliability is currently vibes: pages degrade silently, the watchdog concept (#1148) alerts after the fact, and common failures (collector stopped, feed not pushed, a facet down) wait for a human or an hourly job. The ideal stack declares per-facet freshness SLOs with error budgets on the health page, and the watchdog remediates the known failure classes itself — restart the collector, re-push the feed, re-render config — escalating to Drew only when remediation fails or budget burns.
Design questions to answer
- SLO definitions per facet (feed age, quota read age, orchestration freshness, per-service up): targets, windows, and how budget remaining is computed and displayed.
- Remediation catalogue: which failures are auto-fixable (collector restart, feed re-push, aggregator restart, tunnel restart), with what guardrails, attempt caps, and cooldowns?
- Verification after remediation: how does the watchdog prove the fix worked before standing down, and where is the attempt logged?
- Escalation: what pages Drew (via the PWA/push design) vs lands in NEEDS ORCHESTRATOR, and how does a burning error budget change that threshold?
- Safety: what the watchdog must never do (data writes, deploys, credential changes) and how its own failure is detected.
Acceptance
- Design note landed in repo
doc/+ implementation issues filed and linked as sub-issues of epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184; open questions needing Drew named explicitly in the design note.
Acceptance criteria
- A design note answering every question listed under "Design questions to answer" is landed in the repo
doc/tree and linked from this issue. - Implementation issues are filed and linked as sub-issues of epic epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184 (developing-today/code), each with its own acceptance criteria — this design issue does not itself carry implementation work.
- Every open question that needs Drew is named explicitly in the design note (not buried in prose).
- The remediation catalogue carries per-action guardrails, attempt caps, and cooldowns, and the design names what the watchdog must never do (data writes, deploys, credential changes).
Metadata
Metadata
Assignees
Labels
pm-stack (Pad/Grist) in developing-today/codeMedium: expected under 1 day of workNew feature or capability requestIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Queue or user escalation stateHas a queued task in the Supabase agent queueFiled directly by a humanQueue or user escalation stateQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
developing-today commentedon Oct 27, 2026
CODE-384
dezren39 commentedon Oct 28, 2026
Intake review
Fleet-monitor intake pass (2026-10-28T02:40:00Z): this issue was missing a
priority/*label and/or the standard## Acceptance criteriasection and was not yetmeta/enriched. The criteria section has been appended to the body (original text preserved verbatim above it) and labels normalized.Related: Epic: #1184 (PM stack coherence design series). Siblings in the same series were intake-enriched in prior monitor runs (#1264–#1277).
Notes: Design-series issue (SLOs + self-healing), filed 2026-10-27 by Drew as part of the #1184 batch; body is well-formed (Why + design questions + its own Acceptance section). Still live: merged-PR search this run returns only unrelated PRs whose text coincidentally contains the number — none resolves or implements this design.
Risks: Design-only stall risk (see tail).
Suggested approach: This is one of the #1184 PM-stack coherence design series; its own "Acceptance" section already states the bar (design note in
doc/+ implementation sub-issues), and the appended criteria restate that in the fleet's standard heading so the monitor and queue can gate on it. Risks: design-only items can sit silently — per house rules a design with no implementation link is flagged by the next sweep, so the follow-through that matters is the sub-issue filing, not the note alone. Suggested approach: answer the design questions against the current aggregator/orchestrator code (cite file:line), land the note, file the implementation sub-issues in the same pass.🤖 Agent note · state: handed off
dezren39 commentedon Nov 1, 2026
🤖 Agent note · state: picked up
7 remaining items
dezren39 commentedon Nov 3, 2026
Agent note — picked up / fresh triage (queue task 2521, openchamber-scheduled-lane, 2026-11-03)
Issue remains OPEN with the original design acceptance: a
doc/design note answering five SLO/remediation questions plus implementation sub-issues under epic #1184. Revalidated current state: watchdog detector exists (pm-stack/aggregator/src/watchdog.ts, PR #2431 / issue #1148), but no design note or implementation issues for #1260 exist. The prior task’s blocker was accurate: no bounded implementation fix should be improvised from this design issue.dezren39 commentedon Nov 3, 2026
Agent note — blocked on packet type (2026-11-03)
Fresh triage confirmed the design work remains and is not resolved by the existing watchdog detector. The repo-queue task packet schema allows only
triage,small_fix,pr_shepherd, andci_investigation; it has no design/documentation execution kind, so a correct follow-up packet cannot be enqueued. Blocked withpacket_defectpending queue design-kind support or direct design-workflow routing. No code or design was improvised.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 2521 (triage, blocked, priority 40) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 5, 2026
Specialization filed: #6031 (browser-dependent quota watchdog) is this issue's remediation-catalogue pattern applied to quota sources whose only read path is a logged-in console (Mistral first). Its catalogue entries — CDP session re-validation, browser re-login via standing SSO, escalate-to-browser-agent — with attempt caps (2/rung/incident), disk-persisted cooldowns, and the never-list (no quota-data writes, no credential changes beyond re-storing a session under its registered name, allowlist origins only) are written to fold into this issue's catalogue rather than live as a parallel framework. Escalation split follows this issue's rule: remediation first, Drew only on human-blocked or failed remediation.