Repository navigation
github-backup(fetch): a git fetch that hits the 3600s timeout raises TimeoutExpired out of fetch_mirrors.main() — every later repo is skipped, no failure is recorded, and state.json/MANIFEST.json are never written #3250
Description
Summary
github-backup(fetch): fetch_repo in infra/github-backup/bin/fetch_mirrors.py runs git fetch with timeout=3600 and no TimeoutExpired handling — one hanging fetch raises out of main() and kills the whole run: every later repo is never attempted, no failure is recorded for the timed-out repo, state.json is never written (so the coalesce/fail stamps of repos already fetched in the same run are lost too), fetch.log gets no line, and MANIFEST.json is not rebuilt. The module's own contract — "Failures are appended to fetch.log. A repo failing on 3 distinct days in a row makes the run exit non-zero" — never applies to the slowest, most failure-prone failure mode: a timeout can recur daily forever and never count toward the chronic-failure exit, because it is never recorded as a failure at all.
Evidence (main 8e4ef90e)
fetch_mirrors.py:25-26—subprocess.run(["git", "-C", path, "fetch", "--all", "--prune"], capture_output=True, text=True, timeout=3600, env=env)with notry/exceptanywhere in the file.subprocess.runraisessubprocess.TimeoutExpiredon expiry.fetch_mirrors.py:36-63—main()loops over repos callingfetch_repounguarded;state.jsonis written only after the loop (:52-53), the chronic check and summary print come after that, andmanifest_mod.build(root)is called at:54. An exception anywhere in the loop skips all of it.- Contrast the sibling driver:
clone_all.py:55-60catchessubprocess.TimeoutExpiredexplicitly and returns a recorded failure for that repo, so one repo's timeout never wedges the others. - The same unguarded call reaches the 5-minute drainer:
drain_queue.py:59callsfetch_mirrors.fetch_repo(root, repo, state)with no handling, so a fetch timeout there also raises out ofdrain_queue.main()before its state write (:64-65) and before the remaining queued repos are fetched. (Distinct from github-backup(drain): webhook events are acked even when the fetch failed or the mirror is missing — freshness signal silently dropped #3219, which is about the ack sent after a returned failure; on this path no ack is sent and the crash is the defect.) - Distinct from github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240 (non-atomic state writes / bare
json.load), github-backup(clone_all): a clone killed mid-flight leaves a partial destination every rerun rejects — the 'resumable' driver never resumes that repo #3233 (clone_all partial destination), github-backup(valid_mirror): a bare skeleton with zero refs passes as a verified mirror — clone_all records it ok and MANIFEST.json reports it 'mirrored' #3238 (valid_mirror bareness), github-backup(inventory): a null repositoryOwner is silently read as 'this owner has no repos' — reconcile drops the owner's mirrors from plan.json and MANIFEST.json and exits 0 #3229 (inventory null owner), github-backup: estate mirror system (laptop mirrors + dedicated App + Worker webhooks + weekly sweep) #3215 (umbrella).
Verified (laptop probe, real fetch_mirrors.py)
Probe: ~/git/fleet/probe-2026-10-31-laneB-fetchtimeout/ on the laptop — the real fetch_mirrors.py / ghb_common.py / manifest.py copied verbatim (sha256 f0115270…, 2eb427df…, a489798e…, identical in the sandbox checkout at main 8e4ef90e and the laptop clone), three real bare mirrors, the subprocess layer raising TimeoutExpired for the first repo exactly as git fetch hitting the 3600s timeout does, real git for the rest:
CASE hang-first: main([prog, hang/aaa, good/bbb, good/ccc])
RAISED TimeoutExpired (unhandled): Command '['git', '-C', '.../hang/aaa.git', 'fetch', '--all', '--prune']' timed out after 3600 seconds
fetch attempts: [only hang/aaa — good/bbb and good/ccc never attempted]
state.json exists: False
fetch.log exists: False
MANIFEST.json exists: False
CONTROL no-hang: main([prog, good/bbb, good/ccc])
{"results": {"ok": 2, "fail": 0, "skip": 0, "missing": 0}, "chronic_failures": []}
returned rc = 0
state.json exists: True (both repos stamped last_fetch, fail_days [])
Why it matters
The daily fetch is the freshness mechanism for the whole mirror estate. Mirrors of large/active repos are exactly the ones that can hit a 3600s fetch (slow network, laptop sleep mid-fetch, a wedged remote). Today one such repo silently stops the entire daily pass — every repo after it in plan order goes stale — and leaves no per-repo record, so the chronic-failure alarm that exists precisely to surface a repeatedly failing repo can never fire for it. The only trace is an unhandled traceback in the systemd journal.
Acceptance criteria
- A
git fetchthat exceeds the timeout is recorded as that repo's failure (same shape as a non-zero exit:last_errornoted, today appended tofail_days, afetch.logline written) and the run continues with the remaining repos. -
state.jsonis written andMANIFEST.jsonis rebuilt even when one or more fetches time out; the summary/chronic-failure exit behaviour applies to timeouts exactly as to other failures (3 distinct fail days → exit 1). -
drain_queue.py's fetch path inherits the same behaviour: one repo's timeout does not abort fetching the remaining queued repos or lose the drainer's state write. - Regression tests/probe cover: timeout on the first repo (later repos still fetched, state + log + manifest produced), timeout counted in
fail_days, and the no-timeout control unchanged.
Activity
dezren39 commentedon Oct 31, 2026
🤖 Agent note · filed
dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — agent note: this issue is ADOPTED as a phase of design #3282 ("a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror", design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace), linked as a sub-issue of #3282 and labelled source/design-architect. Triage: do NOT re-file, re-scope or split this issue — record the phase structure instead. Phase mapping: Phase 1 (validity & destinations) = #3238 + #3233; Phase 2 (recorded outcomes & the alarm) = #3250 + #3219 + #3270 + #3257; Phase 3 (durable state & plan retention) = #3240 + #3229 + closing audit #3283 (gated on Phases 1–2). The design records the pins that span the phases: validity = bare + HEAD resolves, with no emptiness exception inside the plan (build_plan excludes zero-commit repos); destinations are absent / valid / skeleton (remove + clone) / debris (quarantine to
.partial-<timestamp>+ clone); every fetch attempt ends in one recorded outcome shared by the daily loop and the drainer, and the drainer acks only onok; the chronic alarm is a streak of 3 consecutive unavailable days (fail, timeout and missing all count); generated state is written atomically only throughghb_commonhelpers; reconcile retains a failed owner's previously planned entries. Where an origin's acceptance left a choice open (the streak-vs-window rule, missing-counts-vs-immediate, report-vs-retain, skeleton handling), the design doc records the decision — triage follows the design's pin, not the open alternative. Each origin keeps its own acceptance criteria and its already-queued triage packet — no new packet was created for it this pass.36 remaining items
dezren39 commentedon Nov 2, 2026
agent-note:v1 — holder: repo-worker-code, at 2026-11-02T13:42:30Z. Parked queue task 5687 for 30 minutes. Fresh metadata confirms #3250 OPEN/assigned to dezren39 and PR #4351 OPEN, non-draft, Drew-authored. The existing queue record identifies the PR as Drew-owned; hard stop respected: no checks, reviews, diff, branch, or worktree inspected and no code touched. Resume only after PR #4351 closes, ownership changes, or explicit rerouting releases the restriction.
dezren39 commentedon Nov 2, 2026
🤖 Agent note · picked up
held/user/owned/do-not-merge/do-not-auto-editlabels on issue github-backup(fetch): a git fetch that hits the 3600s timeout raises TimeoutExpired out of fetch_mirrors.main() — every later repo is skipped, no failure is recorded, and state.json/MANIFEST.json are never written #3250 or PR fix(github-backup): record mirror fetch timeouts #4351 (authorship by the fleet account is not a hard stop — labels are), so I am re-verifying PR fix(github-backup): record mirror fetch timeouts #4351 (diff, checks, Codex findings) and merging if verified.queue/5687-3250-fetch-timeoutdezren39 commentedon Nov 2, 2026
🤖 Agent note · done
7540169c1—fetch_mirrors.fetch_reponow catchessubprocess.TimeoutExpired, records the failure day + timeoutlast_error, and returnsfailso the pass continues, state.json is written, and MANIFEST.json is rebuilt.60e8cb312), branchqueue/5687-3250-fetch-timeoutpython3 infra/github-backup/tests/test_fetch_mirrors.py7/7 OK (incl. the timeout continuation + chronic-failure regression); Codex review clean on both heads (no findings); CodeRabbit pass — its one nitpick (assert the per-repo mirror path in the continuation test) was fixed in60e8cb312and re-reviewed clean. Note: the earlier "Drew-owned" parks on this task were a misread — noheld/user/owned/do-not-*labels exist on issue github-backup(fetch): a git fetch that hits the 3600s timeout raises TimeoutExpired out of fetch_mirrors.main() — every later repo is skipped, no failure is recorded, and state.json/MANIFEST.json are never written #3250 or PR fix(github-backup): record mirror fetch timeouts #4351; hard stops are labels, not authorship.