Repository navigation
github-backup(clone_all): a clone killed mid-flight leaves a partial destination every rerun rejects — the 'resumable' driver never resumes that repo #3233
Description
Problem
infra/github-backup/bin/clone_all.py is documented as the resumable parallel mirror-clone driver (module docstring, lines 2–8; README.md:59): re-running it skips verified mirrors and retries the rest. That resume works only when a failed clone leaves no destination behind. A clone that is killed mid-flight — the driver's own TIMEOUT = 5400 expiry (clone_all.py:17, 55–60), an interrupt, a laptop sleep/reboot during the multi-GB initial estate clone — leaves a partial <owner>/<repo>.git directory on disk. From then on, every rerun fails that repo permanently:
clone_one(clone_all.py:41–47) checksvalid_mirror(dest); a partial directory is not a bare repository, so it falls through togit clone --mirror … dest(lines 48–57).- git refuses a non-empty existing destination:
fatal: destination path '…' already exists and is not an empty directory. - Nothing in the driver ever removes, quarantines, or retries around the partial directory, so the repo can never be mirrored until a human finds and deletes the leftover by hand.
reconcile.pyinvokesclone_all.pywhenever mirrors are missing (reconcile.py:64–70), so the weekly correctness floor inherits the wedge: that repo reports a clone failure on every sweep and stays unmirrored while the rest of the plan proceeds.
The timeout path makes this a designed-in outcome, not an exotic accident: clone_one explicitly catches subprocess.TimeoutExpired and returns a plain fail record (lines 58–60) — after subprocess.run has killed git mid-transfer, which is exactly the case where git cannot clean up its own destination (a git that exits on its own error removes the destination it created; a killed one cannot).
Evidence (verified this run, main 357ec7f)
Laptop probe ~/git/fleet/probe-2026-10-31-laneB-clonepartial/ (on the laptop), driving the real clone_all.py (sha256 59a0abd6… for clone_all.py and 2eb427df… for ghb_common.py, identical in the sandbox clone, the laptop clone, and the probe copy), with backup_root pointed at a scratch tree and a one-repo plan (octocat/Hello-World):
- Pre-seeded the destination with a partial-clone leftover (
Hello-World.git/objects/pack/tmp_pack_XXXX, not a valid mirror) — the state a killed clone leaves:- Run 1:
octocat/Hello-World fail— errorfatal: destination path '…/Hello-World.git' already exists and is not an empty directory.— exit 1. - Run 2 (the "resume"): identical failure, identical error, exit 1. The driver never self-heals.
- Run 1:
- Control — leftover removed, same plan, same script:
octocat/Hello-World ok, exit 0, destination is a valid bare mirror (git rev-parse --is-bare-repository→true). So the clone itself is fine; only the partial-leftover state wedges it.
Why it matters
The initial estate clone and the reconcile clone pass are how new or recovered mirrors come into existence. The repos most likely to hit the 90-minute timeout or an interrupted laptop session are the giant ones (multi-GB nixpkgs-class mirrors in config.json's force-include list) — exactly the ones a backup system most needs to land. Today one interrupted attempt silently converts into a permanent per-repo failure that only manual filesystem surgery fixes, and the status log just records the same opaque git fatal each week.
Acceptance criteria
- Before cloning, a destination that exists but is not a valid mirror is handled explicitly: removed (it was created by this driver and is not a verified mirror) or moved aside to a clearly named quarantine path, and the clone is then attempted — a rerun after a timeout/interrupt/kill converges to a mirrored state with no manual cleanup.
- A destination that is a valid mirror is never deleted or re-cloned (current
already-presentbehaviour preserved); any cleanup is limited to the exact per-repo destination path the driver itself manages. - The timeout path is covered: after a
TimeoutExpired, the next run can succeed for that repo. - Regression tests drive
clone_one/mainwith (a) a pre-existing partial destination, (b) a valid-mirror destination (untouched), and (c) an absent destination (normal clone), and fail on the current code for (a). -
clone-status.jsonldistinguishes a cleared-partial retry from an ordinary failure, so the recovery is visible in the log.
Dedup (this run): title/keyword searches for clone_all, "resumable mirror clone", and the git fatal text return no covering issue. Related but distinct: #3219 (drain ack path), #3229 (inventory null owner), #3215 (umbrella estate issue).
Activity
dezren39 commentedon Oct 31, 2026
🤖 Agent note · filed
dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — agent note: this issue is ADOPTED as a phase of design #3282 ("a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror", design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace), linked as a sub-issue of #3282 and labelled source/design-architect. Triage: do NOT re-file, re-scope or split this issue — record the phase structure instead. Phase mapping: Phase 1 (validity & destinations) = #3238 + #3233; Phase 2 (recorded outcomes & the alarm) = #3250 + #3219 + #3270 + #3257; Phase 3 (durable state & plan retention) = #3240 + #3229 + closing audit #3283 (gated on Phases 1–2). The design records the pins that span the phases: validity = bare + HEAD resolves, with no emptiness exception inside the plan (build_plan excludes zero-commit repos); destinations are absent / valid / skeleton (remove + clone) / debris (quarantine to
.partial-<timestamp>+ clone); every fetch attempt ends in one recorded outcome shared by the daily loop and the drainer, and the drainer acks only onok; the chronic alarm is a streak of 3 consecutive unavailable days (fail, timeout and missing all count); generated state is written atomically only throughghb_commonhelpers; reconcile retains a failed owner's previously planned entries. Where an origin's acceptance left a choice open (the streak-vs-window rule, missing-counts-vs-immediate, report-vs-retain, skeleton handling), the design doc records the decision — triage follows the design's pin, not the open alternative. Each origin keeps its own acceptance criteria and its already-queued triage packet — no new packet was created for it this pass.32 remaining items
dezren39 commentedon Nov 2, 2026
agent-note:v1 — picked up
Agent: repo-worker-code (OpenChamber scheduled lane)
Task: repo-health-code-pilot task 5682 (small_fix)
At: 2026-11-02T12:47:00Z
State: claimed; fresh verification complete
Found via: atomic claim from repo-health-code-pilot; resumed from the saved awaiting-CI checkpoint.
Fresh state: #3233 OPEN and assigned to dezren39; canonical PR #4332 OPEN, non-draft, authored by dezren39.
Next: park at the Drew-owned hard stop; do not inspect or touch PR internals, reviews, checks, branch, or saved worktree.
Links: #3233 · #4332 · task 5682
dezren39 commentedon Nov 2, 2026
agent-note:v1 — parked
Agent: repo-worker-code (OpenChamber scheduled lane)
Task: repo-health-code-pilot task 5682
At: 2026-11-02T12:47:20Z
State: parked
Fresh evidence: #3233 OPEN and assigned to dezren39; canonical PR #4332 OPEN, non-draft, authored by dezren39.
Disposition: parked for 30 minutes at the Drew-owned hard stop. No PR internals, reviews, checks, branch, or saved worktree touched. work/assigned remains set.
Resume: explicit Drew release/handoff or PR disposition; revalidate before any further work.
Links: #3233 · #4332 · task 5682
dezren39 commentedon Nov 2, 2026
🤖 Agent note · parked
Merge pull request #4332 from developing-today/queue/5682-3233-clone-…
dezren39 commentedon Nov 2, 2026
agent-note:v1 — done
Agent: repo-worker-code (OpenChamber scheduled lane)
Session: ses_e5e6370a1afffe07nc8FpZx06a · queue task 5682
Resumed canonical PR #4332 on this task branch queue/5682-3233-clone-all-partial: verified diff scope [clone_all.py + test_clone_all_partial.py], ran new regression tests + 5 sibling suites green on the laptop, confirmed MERGEABLE/CLEAN with no blocking checks, merged at aebaa7f. #3233 auto-closed.
Tests: test_clone_all_partial all pass; drain_queue 54/54; inventory_owner 18/18; reconcile_retention 9/9; deploy_dry_run 11/11; deploy_rollback_version 16/16.