Repository navigation
github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240
Description
Problem
Every generated-state file in the github-backup system is written by truncating the live file and dumping into it, and every reader loads those files with a bare json.load that raises on a torn file. There is no temp-file + rename, no fsync, and no backup copy anywhere in infra/github-backup/bin/ (no os.replace/os.rename/fsync in any of the five scripts).
Writers (all open(path, "w") + json.dump on the live path):
reconcile.py:39—plan.json(the weekly sweep's output, consumed by everything else)fetch_mirrors.py:69—state.json(fetch timestamps / failure history)drain_queue.py:74—state.jsonagain (the 5-minute webhook drainer shares the file with the daily fetch)manifest.py:67—MANIFEST.json, the file the runbook calls the disaster-recovery source of truth
Readers that crash on a torn file:
fetch_mirrors.py:51—json.load(open(state_path))at startup: a tornstate.jsonkills the daily fetch before it fetches anything.drain_queue.py:44and:64— the drainer loadsstate.jsonand (per event)plan.jsonthe same way: one torn write wedges the 5-minute freshness loop too.clone_all.py:24—json.load(open(plan.json))at startup: a tornplan.jsonkills the clone pass.manifest.py:31—json.load(open(plan_path))(theos.path.existsguard does not help): a tornplan.jsonalso kills manifest regeneration, so the DR record stops updating.
The writers and readers run on timers on a laptop that sleeps, reboots, and loses power — the same interruption class as #3233/#3238. A crash, kill, or sleep during any of these writes (they grow with the estate; plan.json/state.json cover every mirror) leaves a truncated JSON file, and from then on every subsequent run of the affected scripts dies at startup with a JSONDecodeError traceback until a human repairs the file by hand. Nothing distinguishes "corrupt state" from any other crash, and the file holds data that cannot be regenerated from the tree alone (state.json's last_fetch coalescing timestamps and fail_days chronic-failure history).
Evidence (verified this run, main 357ec7f)
Laptop probe ~/git/fleet/probe-2026-10-31-laneA-validmirror/ (on the laptop), driving the real scripts (sha256 f0115270… fetch_mirrors.py, 59a0abd6… clone_all.py, a489798e… manifest.py — identical in the sandbox clone, the laptop clone, and the probe copies), with backup_root pointed at a scratch tree:
- Torn
state.json(truncated mid-object, exactly what a killedjson.dumpleaves) →fetch_mirrors.main()crashes:JSONDecodeError: Expecting ',' delimiter: line 1 column 40. The daily fetch never runs. - Torn
plan.json→clone_all.main()crashes:JSONDecodeError: Unterminated string starting at: line 1 column 11. - Torn
plan.json→manifest.build()crashes with the sameJSONDecodeError— MANIFEST.json regeneration is wedged by the same file. - Control — valid
plan.json+state.json:fetch_mirrors.main()runs clean ({"results": {"ok": 0, "fail": 0, "skip": 0, "missing": 1}, "chronic_failures": []}, return 0) and regenerates MANIFEST.json/FAMILIES.md.
Why it matters
This is a backup system: its state files are part of the product, and its whole purpose is surviving interruptions. Today a single ill-timed laptop sleep during a state write silently converts into a persistent outage of the daily fetch, the webhook drainer, and the DR manifest — the failure surfaces only as a traceback in a timer log, and recovery is manual file surgery on files whose history (fetch coalescing, chronic-failure tracking) is lost in the repair. Atomic replace (write temp file in the same directory, fsync, os.replace) is a few lines per writer and makes a torn live file impossible; tolerant loading (keep a last-good copy, or treat a torn file as empty-with-a-loud-warning) is the complementary half.
Acceptance criteria
- All generated-state writes (
plan.json,state.json,MANIFEST.json, and the markdown reports if in scope) are atomic: write to a temp file in the same directory, fsync, thenos.replace— a reader never observes a partial file. - Readers survive a pre-existing torn file instead of dying at startup: on
JSONDecodeErrorthey fall back (empty state / last-good copy) and surface a loud, named warning (log line + non-zero exit or report section), so the corruption is visible rather than silent. - The shared
state.jsonwriters (fetch_mirrors daily + drain_queue every 5 min) cannot interleave into a corrupt file under the atomic scheme (replace is per-writer atomic; any remaining lost-update window between the two writers is either closed or explicitly documented). - Regression tests: (a) torn
state.json→ fetch run proceeds with fallback + warning (fails on current code withJSONDecodeError), (b) tornplan.json→ clone pass and manifest build proceed likewise, (c) a writer killed mid-dump leaves the previous complete file intact, (d) valid files → behaviour unchanged (control passes before and after).
Dedup (this run): title/keyword searches for github-backup + atomic/torn/state.json/plan.json return no covering issue. Related but distinct: #3233 (partial clone destination wedge), #3238 (valid_mirror false positive), #3229 (inventory null owner), #3219 (drain ack path), #3215 (umbrella estate issue).
Activity
dezren39 commentedon Oct 31, 2026
🤖 Agent note · filed
dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — agent note: this issue is ADOPTED as a phase of design #3282 ("a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror", design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace), linked as a sub-issue of #3282 and labelled source/design-architect. Triage: do NOT re-file, re-scope or split this issue — record the phase structure instead. Phase mapping: Phase 1 (validity & destinations) = #3238 + #3233; Phase 2 (recorded outcomes & the alarm) = #3250 + #3219 + #3270 + #3257; Phase 3 (durable state & plan retention) = #3240 + #3229 + closing audit #3283 (gated on Phases 1–2). The design records the pins that span the phases: validity = bare + HEAD resolves, with no emptiness exception inside the plan (build_plan excludes zero-commit repos); destinations are absent / valid / skeleton (remove + clone) / debris (quarantine to
.partial-<timestamp>+ clone); every fetch attempt ends in one recorded outcome shared by the daily loop and the drainer, and the drainer acks only onok; the chronic alarm is a streak of 3 consecutive unavailable days (fail, timeout and missing all count); generated state is written atomically only throughghb_commonhelpers; reconcile retains a failed owner's previously planned entries. Where an origin's acceptance left a choice open (the streak-vs-window rule, missing-counts-vs-immediate, report-vs-retain, skeleton handling), the design doc records the decision — triage follows the design's pin, not the open alternative. Each origin keeps its own acceptance criteria and its already-queued triage packet — no new packet was created for it this pass.27 remaining items
dezren39 commentedon Nov 2, 2026
agent-note:v1 — holder: repo-worker-code (OpenChamber scheduled lane), at=2026-11-02T15:00:35Z. event: parked; task: 5683; queue: repo-health-code-pilot. Per the packet hard gate, #3238 and #3270 remain OPEN; the other prerequisites are CLOSED. Parked for 24 hours with no code/worktree changes. Resume only after both open prerequisites close and fresh validation confirms the full gate.
dezren39 commentedon Nov 3, 2026
🤖 Agent note · parked
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 5683 (small_fix, parked, priority 40) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
OpenCode claimant launched the bound worker session ses_e5e5824dbc1ffe04VFG3rR8OpF for queue task 5683. The worker will use --worker repo-worker-code and finish task 5683 with execution lane=opencode and this session id.
dezren39 commentedon Nov 4, 2026
agent-note:v1
state: parked
task: 5683
queue: repo-health-code-pilot
worker: repo-worker-code
execution: lane=opencode session_id=ses_e5e5824dbc1ffe04VFG3rR8OpF
verified_at: 2026-11-04T18:42Z
Fresh Phase 1/2 gate check for #3240: Phase 1 — #3238 OPEN, #3233 CLOSED. Phase 2 — #3250 CLOSED, #3219 CLOSED, #3270 OPEN, #3257 CLOSED. Per the task's hard gate, parked 24h with no source edits or task worktree changes. Resume only after #3238 and #3270 are both CLOSED and the full gate is freshly revalidated.
Links: #3240 · #3238 · #3270