design(backups): a backup that can prove itself — an expected-set completion contract for the laptop backup jobs (#976) #1854

Relocated from dezren39/darwin-nix#1024 on 2026-10-28 (placement audit): backup design epic for the same jobs as #976/#1026; follows their home. Original body below; issue references inside it are historical (dezren39/nix numbering).

Summary

modules/laptop-backups.nix is the laptop's backup system of record (its header, citing #976 and #1516): two daily jobs, pm-dashboard-backup (local tar.zst + R2 + Drive, each leg sha256-verified) and repo-archive-backup (restic). The machinery works — but a run's success signal does not mean a complete, verified archive exists, and the fleet has now found that six times in the nine days since the dashboard script landed (PR #977):

• id(web): marked spans crossing a soft line break corrupt or fragment on save — single-mark, non-fixpoint #981 — journal cursor advanced on a failed offsite leg (closed)
• id(web): HTML blocks (comments, <details>, <div>) are rewritten as ```html code blocks on save #982 — a run's own retention sweep pruned the archive it had just built (closed)
• Fix /today 500: tolerate string-shaped progressedSinceLastRun #989 — unvalidated DATE argument wrote outside the backup root (closed)
• opencode: pool effort remap — wired rungs, capability order, meta refresh #1010 — a second EXIT trap silently discarded FAILED logging (closed)
• fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021 — prune_remote swallows a failed remote listing: retention silently stops, full success reported (open)
• opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023 — dataset copies unchecked and the manifest generated from whatever staged, so a dropped file is invisible to the script's own verify chain; incomplete archive exits 0 (open)

Three structural causes, all on current main 23551fd:

1. The manifest is descriptive, not normative — modules/pm-dashboard-backup.sh:172-193 builds it by walking the staging tree; verification (:196-219) checks the archive against that same list. Nothing states what a complete backup contains.
2. Failure accounting is per-step and ad hoc — some steps || fail, some tolerate deliberately (journal placeholder, optional-file WARN), some swallow by accident (|| true at :242, unchecked cp -a at :147,150,152,156). Every new step re-decides whether its failure counts.
3. Success leaves no durable record — stdout/journal lines and the journal-window cursor are all a run leaves. fix(cold-store-reaper): curl fallback for S3 downloads urllib cannot deliver #976 (2026-10-27) verified neither timer had ever fired and asked for a freshness check; there is still no artefact such a check could read, and no restore of either job has ever been recorded.

This design is the contract those six issues point at, in five rules:

1. Expected-set manifest — each job declares its expected members (required vs optional) in one place; the manifest records expected vs present vs missing per member; verification compares the archive against the expected set, so a missing required member fails the run before upload, and a missing optional member is recorded in the manifest, not just a WARN.
2. One failure ledger per run — every step's outcome lands in a single ledger; exit status and the success line derive from it at the end. A step is non-fatal only when named as tolerated; an undeclared swallow is a defect.
3. A durable run record — one JSON record per run at a fixed path outside every prune scope: job, date, result, failing steps, archive sha256, per-destination results. Written on complete and failed exits.
4. A freshness watchdog — a third unit beside the jobs reads the run records + both timers' last-trigger times and exits non-zero naming the job when the last complete run is older than a threshold option. It fixes nothing; it makes a stopped backup named, per fix(cold-store-reaper): curl fallback for S3 downloads urllib cannot deliver #976's acceptance.
5. The contract is the adoption unit for fix(cold-store-reaper): curl fallback for S3 downloads urllib cannot deliver #976 coverage — new datasets join by declaring expected members + destinations under this contract, never as another script with a private notion of success.

Phased breakdown

1. Phase 1 — the open instances land → the existing opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023 and fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021 (linked as sub-issues below; fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021 already in progress, opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023 already enqueued). Their regression suites are the seed of the contract's test coverage. No new issue for this phase.
2. Phase 2 — expected-set manifest for pm-dashboard-backup → new sub-issue filed with this design (enqueued): declared required/optional member list; manifest gains expected-vs-present; verify fails on a missing required member; stubbed-harness regression cases for deleted-required (run fails, nothing uploaded) and missing-optional (run completes, manifest records it).
3. Phase 3 — run record + freshness watchdog → new sub-issue filed with this design (enqueued): rule-3 record on every exit path; laptop-backups-watchdog service+timer + threshold option in modules/laptop-backups.nix; watchdog test suite in the repo's established style wired as a flake check. The repo-archive side is covered via timer-firing only — its script and restic credentials live outside this repo (stated honestly in the design doc).

Alternatives considered and why they lose, the full risk analysis (expected-set drift, watchdog false alarms vs Persistent = true catch-up, run-record placement per #982's lesson), and the open questions (scheduled restore drills; whether the repo-archive script is vendored into the repo; eventual restic migration; off-machine alerting; the missing ROADMAP backup row) are in the design doc — they stay questions, not issues, until answered.

Links

• Design doc: files/designs/nix-laptop-backup-completion-contract.md in the fleet goal workspace (keep-the-four-repos-healthy), registered in files/potential-projects.md
• Coverage parent: fix(cold-store-reaper): curl fallback for S3 downloads urllib cannot deliver #976 (this design supplies its "freshness check exists" acceptance mechanism)
• Phase 1 vehicles: opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023, fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021 (sub-issues)
• Phases 2–3: sub-issues filed with this design
• Non-goals: no change to dataset membership, destinations, 30-day retention, or schedule; no opencode-db script changes (lab-ops-hub: container validator returns the raw U-label host for IDN targets — Worker returns the punycode A-label form it judged #659's territory); no credential-handling change; no external monitoring service

🤖 Agent note · state: filed

• Agent: repo-design-architect job
• Session / ID: repo-design-architect cron run 2026-10-28 08:33 CDT (split rotation: opencode / openchamber / nix; nix's first pass)
• Found via: design pass over ROADMAP.md, TODO.md, ARCHITECTURE.md, planning/, docs/ (incl. the 2026-10-19 reliability review), and the open issue list; fix(cold-store-reaper): curl fallback for S3 downloads urllib cannot deliver #976, fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021, opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023 read in full; code evidence verified against the local clone on current main 23551fd
• Task: Design issue — the contract above is the deliverable; phase 1 lands as opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023/fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021, phases 2–3 as their own sub-issues, all enqueued to repo-health-nix-pilot
• Expectation: effort/l across the three phases; this issue closes when all sub-issues land or are explicitly retired

Acceptance criteria

• Phase 1 lands: opencode: pool pass 2 — contributor seats, variant remaps, speed pool #1023 and fix(id-web): degrade unsupported table-cell content instead of panicking on save (#1017) #1021 close with their filed acceptance met.
• Phase 2 lands: a pm-dashboard-backup run with a required dataset member missing or uncopyable fails before any upload, and the manifest of a completed run records every expected member as present or explicitly missing.
• Phase 3 lands: every pm-dashboard-backup run — complete or failed — leaves a JSON run record, and the watchdog exits non-zero naming the job when the last complete run exceeds the threshold.
• No run of either laptop-backups job can print/report full success while a declared-required step failed undeclared.
• This issue closes only when all sub-issues have landed or been explicitly retired.