impl(pm-stack): move stateful services off the laptop and prove host failover

Why

The PM stack still depends on the laptop as the only runtime/data host. Backups reduce data-loss risk but do not make the dashboard available when that host is off. Implement the host-independent deployment and verified cold-failover shape from #1258. Preserve each service's canonical store and the one-writer rule; this is not a big-bang shared-database rewrite.

Scope

Move the aggregator, Kaneo/PostgreSQL, Grist, and Pad runtime to the approved cluster/alternate host as their canonical deployment; keep the laptop as an operator/development client.
Keep aggregator-owned mutable data on the Supabase muse-dev2 path defined by Long-term: migrate the pm stack into Supabase — refactor component state into schemas for central management #3142/aggregator(dashboard): comment API accepts self-asserted author identities — anyone can post as Drew into the orchestrator/GitHub flow (PR #920) #941; retain service-owned Kaneo, Grist, and Pad stores in their approved persistent homes.
Configure persistent storage, backup/restore integration, and health/freshness reporting so no unique record is held only in laptop-local JSON or Docker volumes. Coordinate with pm-stack databases (Kaneo Postgres, Grist, Pad) have no backup routine — docker volumes are the only copy #1514 and pm-stack backups silently produced empty day-dirs for 3 days (stale container names; no alarm) #3869; do not duplicate their backup implementation.
Keep the stable Cloudflare Access URL. Route it to the active host; store host credentials by name in Bitwarden/SOPS and do not commit plaintext secrets.
Implement explicit single-writer fencing and a cold-restore runbook; do not add automatic multi-writer failover.

Acceptance criteria

A documented deployment target and ownership map exists for aggregator, Kaneo, Grist, and Pad; each service has one named canonical data home and one active writer.
The production runtime and persistent data are usable with the laptop powered off; no PM service depends on laptop-only local state to serve or accept durable writes.
Moving the public route to the active host keeps the existing Cloudflare Access hostname and policy; the old writer is fenced before failover promotion.
A cold restore from an off-machine verified backup succeeds for each service-owned store, and an isolated full-stack laptop-off drill exercises real reads and a safe test write.
Measured RTO/RPO are recorded in doc/pm-stack/restore-drills/YYYY-MM.md and meet or explicitly revise the design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258 proposals (4h core RTO; 15m aggregator mutable state; 1h Kaneo/usage/link state; 24h Grist/Pad).
Backup age, active host, restore point, and failed/empty generations are visible in the existing health/watchdog path; no empty directory is reported as a successful backup.
Configuration keeps the development/test path working, and the deployment can be rolled back without losing writes or creating two active writers.

Boundaries

Long-term: migrate the pm stack into Supabase — refactor component state into schemas for central management #3142 owns the phased aggregator schema migration; aggregator(dashboard): comment API accepts self-asserted author identities — anyone can post as Drew into the orchestrator/GitHub flow (PR #920) #941/fix(acp): clear stale cancellation markers on every terminal path (#4618) #4642 own identity/provenance semantics. This issue consumes those contracts, not redesigns them.
pm-stack databases (Kaneo Postgres, Grist, Pad) have no backup routine — docker volumes are the only copy #1514 owns the application-database backup routine; pm-stack backups silently produced empty day-dirs for 3 days (stale container names; no alarm) #3869 owns silent empty-day/container-name failure handling.
pm-stack: make Project #4 auditable — scheduled export of fields, views, item counts, status/area distribution #1218 owns GitHub Project audit export. GitHub remains canonical for GitHub data.
No secrets, copied production data, or raw private records in this repository or drill reports.

Parent: #1184. Design: #1258.

🤖 Agent note · filed

Agent: OpenChamber scheduled lane (repo-worker-code)
Session / ID: queue task 4958; session ses_e5e6716809bffe95LeQ55HH6dQ
Found via: queue claim
Task: Implement the host-independence cutover and prove laptop-off recovery for the PM stack.
Expectation: effort l; finish with service-owned durable homes, single-writer failover, stable Access routing, and a measured restore drill.
Links: design design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258; epic epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184; native parent design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258 (nested under epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184 because epic(dashboard): PM stack coherence - single-URL, fresh data, triage-able, hierarchical #1184 already has 100 direct children)

agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7423 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-j
Session / ID: task 8946
Found via: queue claim
Task: task 8946 for issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 handed off to the opencode execution lane; claimant launches the bound opencode session
Expectation: m — slice 1 only: current-state inventory + ownership map doc under doc/pm-stack/ via branch + PR; no service moves; unruled target/failover/RTO-RPO items listed as open questions for issue design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989, issue design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258, issue pm-stack databases (Kaneo Postgres, Grist, Pad) have no backup routine — docker volumes are the only copy #1514

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-k
Session / ID: task 8946
Found via: queue claim
Task: Re-handing task 8946 (slice 1 of issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 — current-state inventory + ownership map for PM stack services) to the opencode execution lane after the previous handoff's lease lapsed with no session bound; the laptop claimant launches the bound opencode session. Attempt 3/4.
Expectation: effort/m — done looks like a doc under doc/pm-stack/ via branch + PR covering aggregator, Kaneo/PostgreSQL, Grist, and Pad, with unruled target/failover/RTO-RPO items listed as open questions for issue design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258.
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989, issue design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258, issue pm-stack databases (Kaneo Postgres, Grist, Pad) have no backup routine — docker volumes are the only copy #1514; no branch/PR yet

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane H (code-h)
Session / ID: task 8946
Found via: queue claim
Task: Slice 1 (small fix) of issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 for issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 — execution lane assigned opencode; handing off to the laptop claimant, which launches the bound opencode session. NOTE: task 8946 is on its FINAL attempt (4/4) — the bound session should drive it to a terminal state (complete/park/block), not another bare handoff.
Expectation: claimant session works task 8946 to its acceptance criteria at current main.
Links: impl(pm-stack): move stateful services off the laptop and prove host failover #3989

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-a
Session / ID: task 8946
Found via: queue claim
Task: task 8946 (Slice 1 of issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989) assigned to execution lane opencode; handed off to the laptop claimant — this lane does not work it directly
Expectation: claimant launches the bound opencode session; if the lease expires unlaunched, task 8946 returns to the queue
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-z
Session / ID: task 8946
Found via: queue claim
Task: execution lane is opencode — handed to the opencode claimant, which launches a session bound to task 8946 for slice 1 inventory + ownership map on issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989
Expectation: claimant session drives task 8946 to its terminal state
Links: task 8946

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-i
Session / ID: task 8946
Found via: queue claim
Task: task 8946 for issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 handed off to the opencode claimant (assigned_lane=opencode); claimant launches the bound opencode session
Expectation: l — claimant drives task 8946 to a terminal state
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-i
Session / ID: task 8946
Found via: queue claim
Task: handing off task 8946 (small_fix, issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989) to the opencode claimant — execution lane is opencode; claimant launches the bound opencode session. Labels already claimed/in-progress fresh. Attempts 15/16 — CAP RISK: repeated handoffs; if this claimant launch does not complete, the task will need a max_attempts raise.
Expectation: claimant session produces the PM stack current-state inventory for issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-q
Session / ID: task 8946
Found via: queue claim
Task: Slice 1 of issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 (current-state inventory + ownership map for PM stack services) — execution lane assigned opencode; handed to the laptop opencode-claimant for a bound session. Issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 re-verified OPEN fresh this shift, no hard-stop labels; claim labels already in place (queue/claimed, work/assigned, status/in-progress). Note: task 8946 is at attempts 16/16 — if this lease expires unlaunched, the attempts cap is exhausted; claimant throughput is the constraint.
Expectation: effort/m — claimant session produces the doc/pm-stack/ ownership-map doc via branch + PR referencing issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 and issue design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258, and drives task 8946 to a terminal state.
Links: task 8946 · issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 · issue design(pm-stack): durability & host independence — stack state in backed-up durable storage, stack survives the laptop being off #1258

Picked up as task 8946 by lane code-c (repo-queue worker) and handed off to the opencode execution lane per the execution-lane assignment for task 8946. The laptop claimant launches the bound opencode session for task 8946; this lane does not work issue #3989 directly. Note: task 8946 is at attempts 19/32; repeated handoffs — claimant throughput is the constraint.

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-c
Session / ID: task 8946
Found via: queue claim
Task: Slice 1 of issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 — revalidated fresh (open, no hard-stop labels, labels already queue/claimed + status/in-progress); execution lane assigned opencode, handing off to claimant. Attempts 20/32.
Expectation: claimant launches the bound opencode session to drive task 8946 to a terminal state.
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989

🤖 Agent note · handed off

Agent: repo queue worker, lane code-c (repo-worker-code)
Session / ID: cron repo-queue-worker-shift-c run 2026-11-04 21:23 CST; task 8946
Found via: claimed task 8946 this shift; execution-lane assignment returned opencode
Task: Re-claimed task 8946 for issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 (attempt unknown (re-claim)). Fresh check: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 is OPEN, no hard-stop labels (held / user/owned / do-not-merge / do-not-auto-edit), labels already synced to claimed (work/assigned, queue/claimed, status/in-progress). Handed off to the opencode claimant — the claimant launches the bound opencode session; this lane does not work it directly.
Expectation: claimant session drives task 8946 to a terminal state.
Links: task 8946 · issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-s
Session / ID: task 8946
Found via: queue claim
Task: Slice 1 for issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 (current-state inventory + ownership map for PM stack services) — assigned_lane=opencode, handing off to the opencode claimant per the execution-lane protocol; not worked directly by this lane. Fresh check: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989 OPEN, no hard-stop labels, claim labels already in sync.
Expectation: claimant launches the bound opencode session and drives task 8946 to a terminal state.
Links: issue impl(pm-stack): move stateful services off the laptop and prove host failover #3989

agent-note:v1
state: handed off
task: 8946 (repo-health-code-pilot, worker repo-worker-code)
session: ses_e5e5586f1beffetw9oVEKvILNf
found via: queue claim; execution lane opencode
launch: session created on the laptop OpenCode server at verified primary location /home/user/git/code, bound atomically before the prompt was sent.
next: the bound session documents only the verified current-state ownership map for #3989 slice 1; no service moves, deployment changes, writer fencing, or backup/schema implementation. Unruled target/failover/RTO-RPO decisions remain open questions.
links: task 8946 · issue #3989 · design #1258

docs(pm-stack): add current ownership map for #3989

Navigation Menu

Repository navigation

impl(pm-stack): move stateful services off the laptop and prove host failover #3989

Why

Scope

Acceptance criteria

Boundaries

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

impl(pm-stack): move stateful services off the laptop and prove host failover #3989

Description

Why

Scope

Acceptance criteria

Boundaries

Activity

dezren39 commented on Nov 1, 2026

dezren39 commented on Nov 3, 2026

12 remaining items

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions