Repository navigation
Two postmasters on the PM data volume + docker-pm-kaneo start-limit loop — dual Kaneo/PostgreSQL managers (pm-apps oci vs pm-stack compose) #3849
Description
What happened (found 2026-11-01, during the "fix all errors" service sweep)
Kaneo and its PostgreSQL had two competing managers, and for ~10 hours two PostgreSQL postmasters ran against the single pm-stack_postgres_data data volume:
- The pm-stack module's compose path (
pm-kaneo.service+pm-kaneo.timerrunning~/pm-stack/deploy.sh) — live and healthy throughout; the compose postmaster holds Kaneo's connection pool. - The pm-apps module's oci-containers (
docker-pm-kaneo.service,docker-pm-postgres.service) from the code#1752 migration, whose activation never completed: the compose units were never disabled, and the compose self-heal timer resurrected the compose stack at 2026-11-01 00:35, bringing the compose postmaster up alongside the oci postmaster (running since 2026-10-30) on the same PGDATA.
Verified live before intervening: both postmasters accepting connections on the same mounted volume; the oci postmaster had zero established client connections, the compose postmaster had Kaneo's 4 backends. Separately, docker-pm-kaneo.service failed at every activation (Bind for 127.0.0.1:5173 failed: port is already allocated) and restart-looped into start-limit-hit — that failure is what the 2026-11-01 09:30 generation-524 activation surfaced.
PostgreSQL does not support two postmasters on one data directory; that the compose stack kept serving cleanly for 10h is luck (buffer/checkpoint interleaving), not safety.
Interim actions (applied 2026-11-01, verified)
docker-pm-postgres.servicestopped — the idle second postmaster is gone; the compose postmaster is now the only postmaster on the volume.docker-pm-kaneo.servicestopped; both unitsreset-failed. Kaneo never went down (compose stack healthy,GET 127.0.0.1:5173→ 200 before and after)./etc/systemd/systemis read-only on this host, so the units cannot be disabled outside a NixOS switch: if the laptop reboots before the fix below lands via a switch, both units start again. Treat reboot-before-switch as re-opening this incident.
Durable fix
PR #3847 removes the pm-kaneo + pm-postgres declarations from nixos/services/pm-apps, making compose the single declared manager (Grist/Pad/ready-for-agent stay oci; they were never contested). It lands with the next switch (which also carries the flake fold, code#3720).
Decision left open for Drew/fleet
The alternative is completing the #1752 migration for Kaneo/PostgreSQL: disable the pm-stack compose service+timer in the flake and plan the cutover window. PR #3847 deliberately takes the compose branch because that is what has actually been running the PM stack — but the migration intent in #1752 was the opposite, so the call should be explicit. Either way, exactly one manager must survive the next switch.
Related: code#3720 (flake fold), code#1752 (pm-apps migration).
Acceptance criteria
- Exactly one manager owns Kaneo and its PostgreSQL: the losing path's units (pm-apps oci-containers
docker-pm-kaneo/docker-pm-postgres, or the pm-stack compose units if the oci path is chosen instead) are disabled/removed in the NixOS configuration, not merely stopped, so a reboot cannot resurrect the dual-postmaster state. - After a switch (and ideally a reboot or service restart cycle), verification shows exactly one postmaster on the
pm-stack_postgres_datavolume and Kaneo healthy (GET 127.0.0.1:5173→ 200). - The
docker-pm-kaneostart-limit loop is gone (unit removed or fixed), and the interim state recorded in the body is superseded by the final disposition on this issue.
Activity
developing-today commentedon Nov 1, 2026
Recovery complete — Kaneo is back, and the trigger is on the record
Current state (verified 2026-11-01 ~11:08 CST): compose stack healthy —
pm-stack-postgres-1andpm-stack-kaneo-1Up (healthy), Kaneo HTTP 200, live counts 1916 tasks / 1 project / 2 workspaces.docker-pm-kaneoanddocker-pm-postgresremain stopped; PR #3847 removes their declarations at the next switch. A fresh, manifest-verified pm-stack backup of the recovered stack exists (~/backups/pm-stack/2026-11-01/pm-stack-2026-11-01.tar.zst).How the outage happened — including my part in it. The dual-postmaster state itself predates today’s fix run (compose resurrected 2026-11-01 00:35 alongside the oci postmaster from 2026-10-30). At ~10:54 CST, executing this issue’s interim fix, I stopped
docker-pm-postgres.servicewith a plainsystemctl stop. That was the wrong signal for a postmaster sharing a data volume: its clean shutdown wrote a shutdown checkpoint and updated pg_control to its own WAL position (pg_control: "database system was shut down at 2026-11-01 16:54:06 UTC"). The compose postmaster kept serving from memory until it next restarted (~17:01 UTC), then could not recover:invalid checkpoint record / PANIC: could not locate a valid checkpoint record, restart-looping. An immediate (SIGQUIT-class) stop, or no stop until a planned cutover, would not have rewritten pg_control. Recorded here so the runbook learns it: never gracefully stop one of two postmasters sharing a volume.Recovery path taken: stopped the flap (containers + pm-kaneo timer) → bit-for-bit snapshot of the volume (
~/backups/pm-stack/postgres-volume-2026-11-01-pre-recovery/, retained) → validatedpg_resetwalon a scratch-volume copy first (server booted; kaneo DB intact: 46 tables, task count 1916) → appliedpg_resetwalto the real volume → brought the stack up viapm-kaneo.service(deploy.sh) → verified counts + HTTP through the live stack. The Oct 29 logical dump (kaneo.dump) was the fallback and was not needed.Data-loss caveat, stated plainly:
pg_resetwaldiscards WAL past the reset point, so transactions that were committed only in the discarded WAL tail may be lost; the heap itself (what both postmasters flushed) is what was recovered. The PM stack’s GitHub-side reconciliation (projector/aggregator) bounds the practical impact for issue-linked data. No earlier baseline count exists to diff against — the recovered task count is 1916.Also fixed along the way:
~/tools/pm-stack-backup.shhad been dying since 2026-10-30 (the Oct 30/31 and Nov 1 backup day-dirs are EMPTY) — it stilldocker execd the compose-era container namesgrist-grist-1/pad-pad-1, which no longer exist since Grist/Pad became the oci containerspm-grist/pm-pad. Names updated on the laptop (original preserved aspm-stack-backup.sh.orig-2026-11-01); the rerun produced today’s verified archive above. Filed separately for the silent-failure pattern.Reboot caveat stands until PR #3847 lands via a switch: the two oci units are stopped, not disabled (read-only /etc), and would start again at boot.
dezren39 commentedon Nov 2, 2026
Intake review
Related: issue #3869 (the backup gap found during this incident), issue #1752 (the pm-apps migration whose activation never completed — the root of the dual managers).
Notes: Interim mitigation was applied and verified 2026-11-01 (oci postmaster stopped, units reset-failed, Kaneo never down), but the body is explicit that the units cannot be disabled outside a NixOS switch — so the incident is mitigated, not resolved, until the configuration change lands. Already carried priority/high; this pass adds kind/source/effort/status/app and the acceptance section. No merged PR found that dispositions the unit removal, so the issue stays open.
Risks: any laptop reboot before the fix re-arms both managers; two postmasters on one PGDATA risks data corruption that may only surface later.
Suggested approach: decide the surviving manager (compose is the live, healthy one), remove/disable the oci units in the flake, switch, and verify single-postmaster + Kaneo health.
— fleet monitor intake enrichment, 2026-11-02.
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7414 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 4, 2026
agent-note:v1
state: triaged — closing as completed, all acceptance criteria verified live
Triage of task 7414 against fresh state (2026-11-03 ~18:20 CST):
1eb48e878. Merging it was the explicit decision for the compose branch over completing the nixos: bring Grist, Kaneo, Pad, ready-for-agent in as flake services — the last hand-run PM servers #1752 oci migration for Kaneo/PostgreSQL.systemctl list-unit-files "docker-pm-*"returns onlydocker-pm-gristanddocker-pm-pad— thedocker-pm-kaneo/docker-pm-postgresunits no longer exist on the host, so the NixOS switch carrying PR nixos/pm-apps: remove pm-kaneo + pm-postgres oci-containers (compose is their manager) #3847 has landed and a reboot cannot resurrect them.pm-stack-kaneo-1andpm-stack-postgres-1, both Up 2 days (healthy).GET http://127.0.0.1:5173/→ 200.docker-pm-kaneounit is removed, not merely reset-failed.Closing as completed. The interim state in the body is superseded by this disposition.