Repository navigation
github-backup(fetch): fail_days counts every failure day since the last success, not 3 days in a row — intermittent failures weeks apart trip the chronic alarm and latch exit 1 #3257
Description
Summary
github-backup(fetch): the chronic-failure alarm in infra/github-backup/bin/fetch_mirrors.py does not implement its own contract. The module docstring promises: "A repo failing on 3 distinct days in a row makes the run exit non-zero." The implementation instead counts every distinct failure day since the last success, with no consecutiveness check and no expiry: fail_days is only ever appended to (one entry per calendar day) and is only cleared by a successful fetch. A repo that fails once in January, once in February and once in March — never on consecutive days — is declared a chronic failure, and from then on every daily run exits 1 until that repo next succeeds, even on days when nothing failed.
Evidence (main f4f800e7)
fetch_mirrors.py:1-9— module docstring: "Failures are appended to fetch.log. A repo failing on 3 distinct days in a row makes the run exit non-zero (so systemd shows the failure)."fetch_mirrors.py:29-38— on failure,todayis appended tost["fail_days"]if not already present; the list is never pruned, windowed, or checked for consecutiveness. It is reset to[]only on success (:31).fetch_mirrors.py:72-75— the chronic test is purelylen(fail_days) >= 3over the persistedstate.json, andmain()returns 1 if any repo qualifies. Because the list persists instate.json, once the count reaches 3 the exit-1 result latches across all later runs until a success clears it.- Distinct from github-backup(fetch): a git fetch that hits the 3600s timeout raises TimeoutExpired out of fetch_mirrors.main() — every later repo is skipped, no failure is recorded, and state.json/MANIFEST.json are never written #3250 (a fetch timeout raises out of
main()and is never recorded at all — the opposite failure-recording defect), github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240 (non-atomic state writes), github-backup(drain): webhook events are acked even when the fetch failed or the mirror is missing — freshness signal silently dropped #3219 (drainer ack on returned failures). This issue is only about what a recorded failure day counts toward.
Verified (laptop probe, real fetch_mirrors.py)
Probe: ~/git/fleet/probe-2026-10-31-laneA-fetchchronic/ on the laptop — the real fetch_mirrors.py from the laptop clone (sha256 f0115270…, identical to the sandbox checkout at main f4f800e7), a real bare mirror for the valid_mirror sanity check, then the fetch stubbed to fail (rc=1) with the calendar stubbed to three non-consecutive days:
valid_mirror(real bare) = True
2026-01-05 -> fail fail_days = ['2026-01-05']
2026-02-10 -> fail fail_days = ['2026-01-05', '2026-02-10']
2026-03-20 -> fail fail_days = ['2026-01-05', '2026-02-10', '2026-03-20']
chronic_condition(main's exit-1 test) = True
Three failures spaced weeks/months apart — never 3 days in a row — trip the chronic condition.
Why it matters
The exit code is the systemd-visible alarm for the daily mirror fetch. An alarm that fires for intermittent, self-recovering flakiness (a laptop asleep one day in January, a network blip one day in February, a GitHub incident one day in March) and then stays latched trains the operator to ignore it — the exact failure mode a "3 days in a row" streak rule exists to prevent. Conversely the persisted count means the alarm's state depends on ancient history in state.json, not on the repo's current health.
Acceptance criteria
- Chronic means a streak: a repo counts as chronic only when its failures fall on 3 consecutive calendar days (per the docstring), or the docstring and implementation are changed together to a clearly stated window rule (e.g. 3 failures within N days) — code and contract must agree.
- Failure days that fall outside the streak/window no longer count: failures on 2026-01-05, 2026-02-10 and 2026-03-20 do not produce a chronic exit 1.
- A success still resets the streak, and 3 truly consecutive failure days still exit 1.
- Regression tests/probe cover: three non-consecutive failure days (no chronic), three consecutive failure days (chronic), failure–failure–success–failure–failure (no chronic), and the no-failure control unchanged.
Activity
dezren39 commentedon Oct 31, 2026
🤖 Agent note · state: filed
dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — agent note: this issue is ADOPTED as a phase of design #3282 ("a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror", design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace), linked as a sub-issue of #3282 and labelled source/design-architect. Triage: do NOT re-file, re-scope or split this issue — record the phase structure instead. Phase mapping: Phase 1 (validity & destinations) = #3238 + #3233; Phase 2 (recorded outcomes & the alarm) = #3250 + #3219 + #3270 + #3257; Phase 3 (durable state & plan retention) = #3240 + #3229 + closing audit #3283 (gated on Phases 1–2). The design records the pins that span the phases: validity = bare + HEAD resolves, with no emptiness exception inside the plan (build_plan excludes zero-commit repos); destinations are absent / valid / skeleton (remove + clone) / debris (quarantine to
.partial-<timestamp>+ clone); every fetch attempt ends in one recorded outcome shared by the daily loop and the drainer, and the drainer acks only onok; the chronic alarm is a streak of 3 consecutive unavailable days (fail, timeout and missing all count); generated state is written atomically only throughghb_commonhelpers; reconcile retains a failed owner's previously planned entries. Where an origin's acceptance left a choice open (the streak-vs-window rule, missing-counts-vs-immediate, report-vs-retain, skeleton handling), the design doc records the decision — triage follows the design's pin, not the open alternative. Each origin keeps its own acceptance criteria and its already-queued triage packet — no new packet was created for it this pass.7 remaining items
dezren39 commentedon Nov 1, 2026
agent-note:v1 — state=handed_off
worker: repo-worker-code
task_id: 4760
issue: #3257
follow_up_task_id: 5688
summary: Triage confirmed on current main that nonconsecutive failures incorrectly trigger the chronic condition. Follow-up small-fix task 5688 is queued with the consecutive-day contract and four deterministic regression sequences. Triage task 4760 completed as triaged.
verification: isolated, network-free probe confirms spaced failures false-positive; consecutive failures remain chronic; success resets; empty state remains non-chronic.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: picked-up
task: 5688 (repo-health-code-pilot, OpenChamber scheduled lane)
state: Fresh GitHub verification: #3257 is OPEN; no open PR matching #3257 was found; no hard-stop labels are present. The issue is assigned to Drew but only ordinary queue labels were present.
next: Fetch/fast-forward the primary clone, create or resume worktree 5688, inspect the current implementation/tests, then implement the confirmed consecutive-calendar-day streak fix with focused regression tests.
dezren39 commentedon Nov 2, 2026
agent-note:v1
event: done
task: 5688 (repo-health-code-pilot, OpenChamber scheduled lane)
state: Fixed and merged via PR #4334; merge SHA 714289f. Issue #3257 is CLOSED. Streak tracking now keeps only consecutive local-calendar failure dates, preserves success reset, and handles prior cumulative state.
verified: test_fetch_mirrors.py — 6 pass; test_drain_queue.py — 54/54 pass; py_compile and diff check pass. No active CI checks were configured; CodeRabbit was skipped.
bookkeeping: sandbox backlog and sweep.log update deferred to fleet reconciliation as requested.