Design: a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror #3282

Design-architect pass 29 (2026-10-31), audit at origin/main 558cc1d2. Full design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace. NO user/review — every origin carries explicit acceptance with verified probe evidence, and the judgment calls below are the architect's recorded decisions.

Problem

infra/github-backup/ (umbrella #3215, landed today in PR #3214) mirrors Drew's whole GitHub estate — 818 bare mirrors across 17 owners — with a webhook freshness layer, a daily fetch, and a weekly reconcile it calls its correctness floor. In its first day it produced eight defect filings, and every one is a place where the system's claims about itself are not checked against the thing claimed: valid_mirror tests bareness only, so a zero-ref bare skeleton is recorded already-present and reported mirrored in the DR source of truth (#3238), while a non-bare partial destination wedges the "resumable" clone driver permanently (#3233); a fetch timeout raises out of the daily run unrecorded and unwritten (#3250), while the drainer acks webhook events for fetches that failed or mirrors that are missing, deleting the freshness signal (#3219); the chronic-failure alarm counts failure days since the last success instead of the docstring's "3 distinct days in a row" and latches (#3257), yet can never fire at all for a missing mirror (#3270); every generated-state file is written by truncation and read with a bare json.load, so one torn write wedges fetch, drainer and manifest until manual repair (#3240); and a null repositoryOwner is read as "owner has no repos", silently shrinking plan.json and MANIFEST.json with exit 0 (#3229). The sibling precedent is design #1854 ("a backup that can prove itself") for the laptop file-backup jobs — this is that contract's estate-mirror form, in this system's vocabulary; #1854's system is different and untouched.

The rules (recorded decisions)

1. Validity is resolvability. One predicate in ghb_common.py for every consumer (clone skip, fetch missing, manifest status, reconcile missing): bare and HEAD resolves (rev-parse --verify -q HEAD — the call manifest.head_sha already makes). A planned repo is never legitimately empty — build_plan excludes zero-commit repos under the live config (skip_zero_commit_repos: true) — so no emptiness exception exists inside the plan; that settles github-backup(valid_mirror): a bare skeleton with zero refs passes as a verified mirror — clone_all records it ok and MANIFEST.json reports it 'mirrored' #3238's open either/or without an emptiness signal.
2. Destinations have four states, each with a disposition. Absent → clone; valid → skip (never deleted/re-cloned); skeleton (bare, unresolvable HEAD — zero refs, zero objects by construction) → remove and clone (deletion provably loses nothing); debris (exists, not bare — may hold transferred objects) → move aside to <dest>.partial-<timestamp> and clone. Disposition recorded in clone-status.jsonl.
3. Every attempt ends in a recorded outcome. fetch_repo never raises for a per-repo outcome — a timeout becomes a recorded fail naming the timeout (clone_all's existing precedent, generalised). One recorder path turns an outcome (ok/fail/missing) into state + fetch.log + summary, used by both the daily loop and the drainer.
4. An event is acked only when its fetch landed. Drainer acks a planned repo's event only on recorded ok; on fail/missing the event stays in KV for the next drain and the drain_last:: stamp advances only on ok. Unplanned repos keep today's ack + note behaviour (reconcile owns creation).
5. The alarm is a streak of unavailable days. One per-repo day record — fail, timeout-fail and missing all append today; success clears. Chronic = today completes 3 consecutive calendar days (the docstring pinned over github-backup(fetch): fail_days counts every failure day since the last success, not 3 days in a row — intermittent failures weeks apart trip the chronic alarm and latch exit 1 #3257's window alternative; github-backup(fetch): a missing mirror can never trip the chronic alarm — 'missing' returns before any fail_days record, so the daily run exits 0 forever #3270's immediate-vs-3-day choice settled to the same 3-day scale). Legacy fail_days lists are read as this record as-is: non-consecutive legacy days never form a streak, so latched false alarms clear on the first post-upgrade run unless a true streak exists.
6. State is written atomically, loaded tolerantly, only through ghb_common. Temp file in the same directory + fsync + os.replace; last-good copy kept; a torn load falls back with a loud named warning, never a startup traceback. The two state.json writers (daily fetch, 5-minute drainer) read-modify-write under a shared flock lockfile — github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240's lost-update window is closed, not documented.
7. A failed owner keeps its plan. Null repositoryOwner/repositories in a successful GraphQL response is an owner-level error via inventory()'s existing errors channel (present-but-empty still inventories as zero). Reconcile retains the failed owner's previously planned entries in plan.json, marks them retained in the report, and still exits non-zero — github-backup(inventory): a null repositoryOwner is silently read as 'this owner has no repos' — reconcile drops the owner's mirrors from plan.json and MANIFEST.json and exits 0 #3229's report-only alternative is pinned away: there is no drop to make unmistakable.

Phases (adopt the origins — no parallels)

• Phase 1 — Validity and destinations: github-backup(valid_mirror): a bare skeleton with zero refs passes as a verified mirror — clone_all records it ok and MANIFEST.json reports it 'mirrored' #3238 + github-backup(clone_all): a clone killed mid-flight leaves a partial destination every rerun rejects — the 'resumable' driver never resumes that repo #3233. Lands first: every other phase's missing depends on what validity means.
• Phase 2 — Recorded outcomes and the alarm: github-backup(fetch): a git fetch that hits the 3600s timeout raises TimeoutExpired out of fetch_mirrors.main() — every later repo is skipped, no failure is recorded, and state.json/MANIFEST.json are never written #3250 + github-backup(drain): webhook events are acked even when the fetch failed or the mirror is missing — freshness signal silently dropped #3219 + github-backup(fetch): a missing mirror can never trip the chronic alarm — 'missing' returns before any fail_days record, so the daily run exits 0 forever #3270 + github-backup(fetch): fail_days counts every failure day since the last success, not 3 days in a row — intermittent failures weeks apart trip the chronic alarm and latch exit 1 #3257. Independent of Phase 1 in code; its missing semantics inherit Phase 1's predicate whenever that has landed.
• Phase 3 — Durable state and a plan that cannot shrink silently: github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240 + github-backup(inventory): a null repositoryOwner is silently read as 'this owner has no repos' — reconcile drops the owner's mirrors from plan.json and MANIFEST.json and exits 0 #3229 + a new closing audit issue filed with this design (gated on Phases 1–2, status/blocked): helper adoption across all five scripts, the closing inventory of remaining bare json.load/open("w") on generated state, the contract stated in README + RUNBOOK, evidence on github-backup: estate mirror system (laptop mirrors + dedicated App + Worker webhooks + weekly sweep) #3215.

Alternatives rejected (in the doc)

Fixing the eight independently (the defects are the divergences — that is how the family arose); the window rule for the alarm (the docstring is the operator contract systemd is judged against; the streak is also the less latch-prone reading); deleting debris outright (its objects may be the only copy mid-estate-clone; quarantine is cheap); fetching into the skeleton in place (re-derives clone --mirror's remote/refspec setup in a second place; removal is lossless by the state's definition); a separate watchdog for missing mirrors (the fetch exit code is this system's stated alarm channel — #1854's watchdog belongs to a system without one); report-only plan shrinkage (MANIFEST is regenerated from the plan, so a shrunken plan falsifies the DR record for mirrors that exist).

Risks (in the doc)

Phase 1 recounts skeletons as missing on the first post-landing reconcile/fetch while #3215's initial 818-mirror clone is still in flight — the correction working, but triage should announce the one-time recount on #3215. Ack retention grows the Worker KV queue for persistently failing repos (bounded by coalescing + the rule-5 alarm). Plan retention keeps a vanished owner's repos planned while it errors — loudly (weekly report marks them, exit stays 1), resolved via config exclusions/gaps where owner rulings already live.

Open questions (in the doc, never as issues)

A third MANIFEST status for the skeleton state vs folding into missing; the 3600s fetch timeout against the drainer's 5-minute cadence (decide on Phase 2's timeout records); state.json schema versioning; Worker-side dead-lettering; FAMILIES.md / reconcile-report atomicity (Phase 3's audit dispositions them explicitly).

Acceptance criteria

• Phase 1 lands: github-backup(valid_mirror): a bare skeleton with zero refs passes as a verified mirror — clone_all records it ok and MANIFEST.json reports it 'mirrored' #3238 and github-backup(clone_all): a clone killed mid-flight leaves a partial destination every rerun rejects — the 'resumable' driver never resumes that repo #3233 close with their filed acceptance met, under rules 1–2 as recorded here.
• Phase 2 lands: github-backup(fetch): a git fetch that hits the 3600s timeout raises TimeoutExpired out of fetch_mirrors.main() — every later repo is skipped, no failure is recorded, and state.json/MANIFEST.json are never written #3250, github-backup(drain): webhook events are acked even when the fetch failed or the mirror is missing — freshness signal silently dropped #3219, github-backup(fetch): a missing mirror can never trip the chronic alarm — 'missing' returns before any fail_days record, so the daily run exits 0 forever #3270 and github-backup(fetch): fail_days counts every failure day since the last success, not 3 days in a row — intermittent failures weeks apart trip the chronic alarm and latch exit 1 #3257 close with their filed acceptance met, under rules 3–5 as recorded here.
• Phase 3 lands: github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240 and github-backup(inventory): a null repositoryOwner is silently read as 'this owner has no repos' — reconcile drops the owner's mirrors from plan.json and MANIFEST.json and exits 0 #3229 close with their filed acceptance met, under rules 6–7 as recorded here, and the closing audit issue completes its inventory + contract statement.
• No surface of the system (clone-status, fetch summary/exit code, drain summary, MANIFEST.json, reconcile report) can report coverage, success or health that the underlying mirrors, records or inventory do not support.