Repository navigation
github-backup(inventory): a null repositoryOwner is silently read as 'this owner has no repos' — reconcile drops the owner's mirrors from plan.json and MANIFEST.json and exits 0 #3229
Description
github-backup(inventory): a null repositoryOwner is silently read as "this owner has no repos" — reconcile then drops the owner's mirrors from plan.json and MANIFEST.json and exits 0
Evidence (verified this run, 2026-10-31)
infra/github-backup/bin/ghb_common.py, inventory_owner():
data = json.loads(r.stdout)
conn = (data.get("data", {}).get("repositoryOwner") or {}).get("repositories")
if conn is None:
return reposAny successful (gh exit 0) GraphQL response whose repositoryOwner is null — the owner was renamed, the token lost visibility of the org (e.g. SAML enforcement, the exact failure mode config.json already records as a GAP for vertexinc), or the org is otherwise not visible — returns an empty repo list for that owner, indistinguishable from an owner that genuinely has zero repositories. The errors channel in inventory() is never populated for this case.
Laptop probe (real ghb_common.py from main 7d2f1f77, sha256 2eb427df… matching the sandbox clone; run() stubbed to return canned GraphQL payloads; probe dir ~/git/fleet/probe-2026-10-31-laneA-ghb-inventory/ on the laptop):
control-visible: repos=['some-owner/repo1'] errors={}
null-owner: repos=[] errors={} <- the defect
gh-failure: repos=[] errors={'some-owner': 'gh api graphql failed for some-owner: boom'}
null-repositories: repos=[] errors={} <- same defect, repositories field null
The control cases behave: a visible owner inventories normally, and a gh CLI failure raises and is recorded per-owner. Only the null-owner/null-repositories shapes are swallowed.
Why it matters
reconcile.py is the system's correctness floor (its own docstring: inventory errors per owner are reported, and "Exit code is non-zero only on owner-level inventory failure, so the timer unit surfaces it"). With this defect, one invisible owner causes the weekly reconcile to:
- overwrite
plan.jsonwith none of that owner's repos (they surface only as "no longer planned (mirrors kept on disk)", which reads as an intentional removal); - regenerate
MANIFEST.jsonfrom the shrunken plan —manifest.pyiterates the plan only — so the disaster-recovery source of truth silently stops recording mirrors that still exist on disk; - exit
0(return 1 if errors else 0, anderrorsis empty), so the systemd unit reports success.
A backup system whose inventory layer cannot distinguish "owner has nothing" from "owner could not be seen" will eventually un-plan a live part of the estate and call it success.
Distinct from: #3219 (the drainer's ack path), #3215 (the umbrella system issue, which tracks the system as a whole, not this defect).
Acceptance criteria
inventory_owner()treats a null/missingrepositoryOwner(and a null/missingrepositoriesconnection) in an otherwise successful GraphQL response as an owner-level error: it raises (or otherwise records an error for that owner ininventory()), naming the owner and the shape received.- A legitimately empty owner (connection present, zero nodes, no next page) still inventories as zero repos with no error.
- Reconcile surfaces that failure the way its docstring promises: the owner appears in the report's inventory-errors section and the run exits non-zero, without overwriting
plan.jsonin a way that drops the failed owner's previously planned repos (or, if plan.json is still written, the report makes the drop's cause unmistakable). - Regression tests cover: null
repositoryOwner, nullrepositories, a GraphQL response carrying a top-levelerrorsfield, and the empty-but-visible owner control.
Activity
dezren39 commentedon Oct 31, 2026
🤖 Agent note · filed
dezren39 commentedon Oct 31, 2026
Intake review
Related: #3215 (github-backup estate mirror system — the parent system this inventory feeds), #3219 and #3230 (sibling github-backup / cold-store verification gaps filed the same day).
Notes: Verified live by the filer with a laptop probe against real
ghb_common.py(main7d2f1f77): the null-owner and null-repositories shapes returnrepos=[]with an empty errors channel, indistinguishable from a genuinely empty owner, while a hardghfailure is recorded per-owner. The issue already carries acceptance criteria in its body (probe cases), so no body change was made.Risks: This is the fleet's backup correctness floor: a silently-dropped owner loses its mirrors from plan.json/MANIFEST.json with exit 0 — a data-loss-adjacent failure that only surfaces at restore time. Priority/normal stands, but the fix should land before the next reconcile cycle is trusted.
Suggested approach: Treat null
repositoryOwner/ nullrepositoriesas a per-owner inventory error (populate the existing errors channel, non-zero overall result), and pin the probe's four cases (visible / null-owner / gh-failure / null-repositories) as tests.— fleet monitor intake, 2026-10-31 fleet-monitor run (17:12 CDT)
dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — agent note: this issue is ADOPTED as a phase of design #3282 ("a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror", design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace), linked as a sub-issue of #3282 and labelled source/design-architect. Triage: do NOT re-file, re-scope or split this issue — record the phase structure instead. Phase mapping: Phase 1 (validity & destinations) = #3238 + #3233; Phase 2 (recorded outcomes & the alarm) = #3250 + #3219 + #3270 + #3257; Phase 3 (durable state & plan retention) = #3240 + #3229 + closing audit #3283 (gated on Phases 1–2). The design records the pins that span the phases: validity = bare + HEAD resolves, with no emptiness exception inside the plan (build_plan excludes zero-commit repos); destinations are absent / valid / skeleton (remove + clone) / debris (quarantine to
.partial-<timestamp>+ clone); every fetch attempt ends in one recorded outcome shared by the daily loop and the drainer, and the drainer acks only onok; the chronic alarm is a streak of 3 consecutive unavailable days (fail, timeout and missing all count); generated state is written atomically only throughghb_commonhelpers; reconcile retains a failed owner's previously planned entries. Where an origin's acceptance left a choice open (the streak-vs-window rule, missing-counts-vs-immediate, report-vs-retain, skeleton handling), the design doc records the decision — triage follows the design's pin, not the open alternative. Each origin keeps its own acceptance criteria and its already-queued triage packet — no new packet was created for it this pass.