pm-stack backups silently produced empty day-dirs for 3 days (stale container names; no alarm)

pm-stack backups silently produced empty day-dirs for 3 days

~/backups/pm-stack/ has day directories for 2026-10-30, 2026-10-31, and 2026-11-01 that were created on schedule but contain no archive — the last good tarball before today was 2026-10-29. Nothing alarmed; the gap was found by hand during the 2026-11-01 Kaneo/PostgreSQL incident (#3849), where the Oct 29 kaneo.dump turned out to be the only logical fallback.

Cause: ~/tools/pm-stack-backup.sh creates the day dir up front, then set -e kills it mid-run, leaving the empty dir as camouflage. The mid-run deaths were stale container names from the compose→oci migrations: docker exec pm-stack-postgres-1 failed while the compose stack was down for the pm-apps activation (Oct 30), and after Grist/Pad became oci containers the script still targeted grist-grist-1 / pad-pad-1 (live names: pm-grist / pm-pad).

Already fixed on the laptop (2026-11-01): names updated in ~/tools/pm-stack-backup.sh (original kept as pm-stack-backup.sh.orig-2026-11-01), and a rerun produced a verified archive for 2026-11-01 (manifest-checked by the script itself, sha256 d7a8881df26333269d4cb9b732b3daafbb5a6fdf8a7ee18a084d1baae7551833).

What remains worth doing:

The script (or its scheduling service, code#1516) should treat "day dir exists but no verified archive" as a failure worth surfacing — an empty dir currently reads as a completed backup day in any casual listing.
Container names are migration-fragile; resolving targets by label/project (or failing with the container list in the log) would have made this a one-day, self-explaining failure instead of a silent 3-day gap.

Acceptance criteria

A backup run that ends without a verified archive is surfaced as a failure (alarm, non-zero service result, or filed issue) — an empty day-dir no longer reads as a completed backup day.
Backup targets are resolved robustly against container renames (by label/project, or the run fails loudly with the actual container list in the log), so a compose-to-oci migration cannot silently break targeting again.
A drill or log excerpt demonstrates the failure path firing on a simulated missing-container run.

Intake review

Related: issue #3849 (the incident during which this gap was found; its Oct 29 kaneo.dump was the only logical fallback), issue #1516 (the backup scheduling service named in the body).

Notes: The immediate cause (stale container names in ~/tools/pm-stack-backup.sh) was already fixed on the laptop 2026-11-01 and a verified 2026-11-01 archive exists — what remains, and what this issue now tracks, is the silent-failure mode itself: set -e death after the day-dir is created, and migration-fragile container names. Already carried priority/normal; this pass adds kind/source/effort/status/app and the acceptance section.

Risks: backups that report success while empty are worse than no backups — any restore plan citing the Oct 30–Nov 1 window must treat those days as absent.

Suggested approach: make the script write the day-dir only after a verified archive exists (or rename it to a .failed marker on death), and resolve containers via compose/oci labels with a loud failure listing what was found.

— fleet monitor intake enrichment, 2026-11-02.

agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03

🤖 Agent note · handed off

Agent: fleet monitor (repo-fleet-monitor)
Session / ID: task 8296 · repo-fleet-monitor run 2026-11-03 18:12 CST
Found via: monitor reconciliation
Task: Claim evaporated — task 8296 returned to queued and unclaimed in repo-health-code-pilot after the holding run ended without a terminal verb (checkpoint stage: handed_off). Labels reset to the truth: queue/queued + status/needs-triage. The task cap was also raised this run where it had sat dead-in-queue, so it is claimable again.
Expectation: next claimant picks up from the checkpoint stage named above.

agent-note:v1
state: picked-up
queue: repo-health-code-pilot
task: 8296 (triage, worker repo-worker-code, lane 8ac61616-3815-4e7b-b33d-840a3829154a)
by: openchamber-scheduled-lane
at: 2026-11-04T02:12:00Z

Resumed from the handed_off checkpoint. Prior handoff to the opencode lane is stale (the fleet monitor reset the claim to queued), so this lane is the active owner. Triaging at current main now.

agent-note:v1
state: triaged
queue: repo-health-code-pilot
task: 8296 (triage, worker repo-worker-code, lane 8ac61616-3815-4e7b-b33d-840a3829154a)
by: openchamber-scheduled-lane
at: 2026-11-04T02:13:00Z

Verdict: STILL LIVE, scoped, no duplicate PR.

Evidence (~/tools/pm-stack-backup.sh, read at current laptop state):

Line 12 set -euo pipefail.
Line 22 mkdir -p "$STAGE/kaneo" "$STAGE/grist/docs" "$STAGE/pad" "$OUTDIR" creates the day directory before any dump runs.
Line 21 trap 'rm -rf "$STAGE"' EXIT cleans only the staging dir; nothing removes $OUTDIR on failure.
Lines ~24-80 run the dumps (docker exec / docker cp). Any failure there exits under set -e and leaves an empty $OUTDIR, which is the reported silent failure.
The verify block (ends ~line 33) only guards the archive path after it exists, so it does not cover the empty-dir case.

Already fixed per the 2026-11-01 note: the stale container names. Remaining gap: the day-dir-first ordering and the missing failure cleanup.

Scope of the fix (for a follow-up small_fix; not implemented in this run):

Create $OUTDIR only after the archive is staged, or add a failure trap that removes an empty $OUTDIR (or renames it to $OUTDIR.failed).
Resolve pm-stack containers via labels with a loud failure that lists what was found, instead of fixed names.
Add a check that fails the run when a day directory is empty.

Blockers/ordering: the script is outside the repo (~/tools/), so the fix lands via the laptop tooling and is not a developing-today/code PR. No held or user-owned labels on this issue. Nothing was changed in this run.

Found via: queue task 8296 (triage)

agent-note:v1
state: done
queue: repo-health-code-pilot
task: 8296 (triage, worker repo-worker-code, lane 8ac61616-3815-4e7b-b33d-840a3829154a)
by: openchamber-scheduled-lane
at: 2026-11-04T02:16:00Z

Triage completed and queue task closed as triaged. Findings are in comment 6620672788. No separate follow-up task was enqueued because the fix target is the tracked ~/tools/pm-stack-backup.sh in the hatch-personal checkout, outside this developing-today/code queue's worktree/PR scope; the blocker and fix outline are recorded on this issue. Claim labels removed. No code changed.

Navigation Menu

Repository navigation

pm-stack backups silently produced empty day-dirs for 3 days (stale container names; no alarm) #3869