Repository navigation
feat(aggregator): alert throat — sentinel incident intake, Supabase ledger, Web Push delivery + ack #3676
Description
Build the alert throat (Drew approved 2026-11-01, main chat): every sentinel posts incidents to ONE intake; one engine owns dedupe, delivery, escalation, and ack; every transition lands in a durable ledger.
Drew's amendments to the original design (#3450):
- Delivery uses the feat(aggregator): installable PWA + push notifications — NEEDS DREW and quota walls reach Drew's phone within a minute #1254 PWA Web Push design (doc/2026-11-01T03-54-00Z_design_pwa_push_notifications.md, PR doc: PWA + push notifications design for #1254 (epic #1184) #3314) — this issue builds the push plumbing of feat(aggregator): Web Push plumbing — VAPID, subscription store + API, service worker (#1254 slice b) #3316 plus an
incidentnotification class, ahead of the dashboard-surface classes in feat(aggregator): notification engine — class triggers, dedupe, quiet hours, escalation (#1254 slice c) #3317. - The ledger lives in Supabase muse-dev2, not local state files: schema
alertsis already created —incidents(with a partial unique index on (source, incident_key) for non-closed rows),incident_events,push_subscriptions,notification_sends. - It ties into the PM stack: intake, engine, and the ack surface are aggregator features; incidents surface on a new
/alertspage.
Already done:
- Supabase
alertsschema (above) in muse-dev2. - VAPID keypair generated; private key in Bitwarden Secrets Manager as
pm-aggregator.vapid-private-key; public key ships in the page/worker. VAPID subject: mailto:drewrypope@gmail.com (design Q4 default).
Scope:
- PWA installability per feat(aggregator): PWA installability — manifest, icons, install affordance (#1254 slice a) #3315 (manifest, icons, meta;
PM_AGG_PUBLIC_ORIGINdefaulting to the Tailscale Serve HTTPS origin — verify serve 443→4870 on the laptop). - Real service worker per feat(aggregator): Web Push plumbing — VAPID, subscription store + API, service worker (#1254 slice b) #3316 with push + notificationclick handlers and eviction-on-activate for foreign registrations (design §6 fallback path; no probe data exists yet), plus
/api/sw-probereporting. - Subscription API backed by Supabase
alerts.push_subscriptions(per-class map + quiet prefs round-trip; 404/410 gone-count pruning). - Incident intake
POST /api/alerts(bearer token in Bitwarden aspm-aggregator.alerts-intake-token): {source, key, severity, title, body}; dedupe on (source, key); every post writes incident +incident_eventsrows. - Engine tick: new critical incident → Web Push to subscriptions with the
incidentclass on,delivered_at+ send log; unacked critical after the ack window (config, default 15 min) → exactly one escalation push. Incident class delivers immediately (it is the breakthrough class — the sentinels are external, so the feat(aggregator): notification engine — class triggers, dedupe, quiet hours, escalation (#1254 slice c) #3317 §2 "engine is down" reasoning does not apply); dashboard classes keep the design's quiet-hours rules. /alertspage: open incidents, ack (exactly once; second ack is a no-op) and close actions, history from Supabase. Push payload deep-links to the incident on this page.- A
scripts/post-alert.tsCLI in the aggregator for sources to post incidents (first wired source: the Globalping watch failure path; second: manual seed for acceptance).
Acceptance: subscription round-trips through Supabase; a seeded critical pushes to a real subscription; a duplicate post sends no second push; escalation fires exactly once inside a shortened test window; ack stops the ladder and is exactly-once; all transitions present in alerts.incident_events; no secret material in repo, store, or logs.
Acceptance criteria
- PWA installability landed per feat(aggregator): PWA installability — manifest, icons, install affordance (#1254 slice a) #3315: manifest, icons, and meta in place;
PM_AGG_PUBLIC_ORIGINdefaults to the Tailscale Serve HTTPS origin. - Service worker per feat(aggregator): Web Push plumbing — VAPID, subscription store + API, service worker (#1254 slice b) #3316 serves push and notificationclick handlers, evicts foreign registrations on activate, and
/api/sw-probereports registration state. - Subscription API is backed by Supabase
alerts.push_subscriptions: per-class map and quiet preferences round-trip; 404/410 responses prune dead subscriptions. - Incident intake
POST /api/alertsaccepts sentinel incidents (bearer auth), dedupes on (source, incident_key) for non-closed rows in thealertsschema, and records every transition inincident_events. - Incidents surface on the aggregator
/alertspage with ack/escalation state matching the ledger. - The
incidentnotification class delivers via the feat(aggregator): installable PWA + push notifications — NEEDS DREW and quota walls reach Drew's phone within a minute #1254 Web Push design ahead of the feat(aggregator): notification engine — class triggers, dedupe, quiet hours, escalation (#1254 slice c) #3317 dashboard-surface classes.
Activity
developing-today commentedon Nov 1, 2026
CODE-1499
feat(aggregator): alert throat — sentinel intake, Supabase ledger, We…
test(aggregator): bring the manifest/a11y suites along with the Alert…
dezren39 commentedon Nov 1, 2026
Intake review — fleet monitor (2026-11-01)
Related: design discussion in #3450; delivery design #1254 / PR #3314; build prerequisites #3315 (PWA installability), #3316 (service worker + push plumbing), follow-on classes in #3317. Ledger schema
alertsalready exists in Supabase muse-dev2 per the issue body.Notes: this is a large multi-part epic-style build (effort/l, priority/high already set). The scope list in the body is the spec; acceptance criteria appended above are drawn from it.
Risks: Web Push subscription churn (404/410 pruning) and foreign service-worker registrations are the known sharp edges — both are named in the scope. VAPID private key and the intake bearer token live in Bitwarden Secrets Manager; neither belongs in code or comments.
Suggested approach: land in scope order (installability → service worker → subscription API → intake → /alerts page), parking the queue task at each verified stage so the ledger work can be reviewed independently.
Merge pull request #3680 from developing-today/feat/alert-throat-3676
dezren39 commentedon Nov 1, 2026
Built, deployed, and verified end-to-end (2026-11-01). PRs: #3680 (the alert throat), plus #3679 — a prerequisite emergency fix: main itself did not boot (
isUnmatchedChargelost its closing brace in the #2445 merge; the aggregator only survived because the running process predated it).Deployed: primary clone at the merge of #3680,
pm-aggregatorrestarted (abun installin the primary clone was needed for the newweb-pushdependency — worth adding to the deploy checklist for dependency-changing PRs). Tailscale Serve verified:https://amd-6.raptor-pumpkinseed.ts.net(443) →127.0.0.1:4870, unchanged.Acceptance evidence (test incident
97d7d6a4-5c35-47a4-8283-530a17542329, sourceacceptance-3676, against the production service + the real Supabase ledger):/attention, standalone, icons incl. maskable);/sw.jsat the origin is the real worker (push + notificationclick + §6-fallback cache purge on activate; the probe script also unregisters foreign registrations page-side). Tokenless intake → 401.alerts.push_subscriptionswith the exact per-class map + quiet prefs → DELETE → row gone, repeat DELETE 404s.scripts/post-alert.ts; the engine tick pushed it — the receiving endpoint logged an RFC 8030 POST with VAPID authorization,content-encoding: aes128gcm, TTL 3600, andalerts.notification_sendsrecordedoutcome=sentwithdelivered_atset.created:false, occurrences 1→2, no second push (sends count unchanged),duplicate_suppressedevent written.ALERT_ACK_WINDOW_MIN=1(temporarily, via the user-manager environment; removed and the service restarted on defaults afterward): exactly one escalation push,escalated_atset, no third send across many later ticks.acted_on_at), second ack returnedtransitioned:falseand wrote nothing. Close transitioned once.opened -> delivered -> duplicate_suppressed -> escalated -> acked -> closed.pm-aggregator.alerts-intake-token; the VAPID private key is fetched by name at runtime — no secret material in repo, store, or logs.One step remains, and it is a phone step, not code: a vendor push subscription (FCM/Apple/Mozilla registration) could not be created from any laptop browser — Helium registers the worker and takes the permission grant but its push registration fails ("push service error", de-Googled build with no FCM client), and the stock Chromium/Brave sessions wedged on new-tab navigation in this desktop session. So delivery was proven against a standards-compliant local push endpoint (real P-256 subscription keys, real VAPID + aes128gcm on the wire) instead of a phone. To finish: on Drew's phone, open the origin, install the PWA (required for iOS push), open
/alerts, tap Enable push on this browser — the subscription lands inalerts.push_subscriptionsand every downstream behavior is the proven path above.Also landed here: #3315's installability and the core of #3316's plumbing (those issues stay open for their remaining acceptance items — #3316's branch-B Access deep-link check is still gated on #1182). First wired source per scope item 7 is the CLI itself; pointing the Globalping watch's failure path at
post-alert.tsis the follow-up.fix(aggregator): alerts push UX — permission flow, quiet-hours off, t…
dezren39 commentedon Nov 1, 2026
Push UX fix shipped — PR #3707, merged 2026-11-01T13:48:55Z, deployed (primary ff + pm-aggregator restart, healthy).
Drew's phone report, six defects → fixes:
display: standalone, 192/512 + maskable icons, apple-touch-icon + apple-mobile-web-app-capable all served at the tailnet origin.notification permission: <raw value>, alert()ed by the page. The client now throws coded errors only; the page renders plain words per state (blocked → device-settings steps; prompt dismissed → tap again and choose Allow). The permission request still runs inside the Enable tap, before subscribing.enabled:falsein the existing quiet jsonb (no migration);inQuietHoursreturns false when disabled; time inputs disable while off.POST /api/push/testsends a real push (classtest) to all subscriptions +alerts.notification_sendsrows; plain-words result on the page.Acceptance evidence (production, after deploy):
inQuietHoursdisabled, the test payload, andsendTestPushsends-row recording incl. the zero-subscription case.POST /api/push/teston production →{subscriptions: 1, sent: 1, gone: 0, errors: 0}against Drew's live phone subscription; ledger rowclass=test, outcome=sentat 2026-11-01T13:49:52Z. (Pre-merge, the same path was proven against the /tmp/mockpush RFC 8030 receiver: VAPID + aes128gcm POST logged,sentrow at 13:48:13Z; the zero-subscription response is all-zeros, which the page renders as "No subscriptions yet — enable push first".)enabled: true, which absent already meant).fix(pm-aggregator): /alerts push panel — judge support from the windo…
Merge pull request #3745 from developing-today/fix/alerts-push-panel-…