Repository navigation
cold-store-reaper: state.json and credentials.json are written in place with O_TRUNC — one torn write wedges every later run with a raw JSONDecodeError #3279
Description
Problem
ColdStore._write_private_json (pkgs/cold-store-reaper/cold_store.py:373-378) writes credentials.json and state.json in place: os.open(path, O_WRONLY|O_CREAT|O_TRUNC) followed by json.dump directly into the destination file. There is no temp file, no os.replace, and no fsync. A crash, power loss, OOM kill, or any exception mid-json.dump leaves the destination truncated to a partial JSON document.
Both readers then fail hard with a raw error, not the module's own error type:
load_state(cold_store.py:393-396) does a barejson.loads(self.state_path.read_text())— a tornstate.jsonraisesjson.decoder.JSONDecodeErroruncaught.load_stateis on the path ofensure_folders(:687) and_folder_id/_d1_endpoint(:709, :1242), so one torn write wedges every later run until a human hand-repairs the file.load_credentials(cold_store.py:381-391) does the same barejson.loads— a torncredentials.jsonraises rawJSONDecodeErrorinstead of theColdStoreError("credentials.json is missing …")-style actionable errors the function otherwise raises, masking the real cause (the file exists; it is half-written).
save_state (cold_store.py:398-405) is called on routine paths (e.g. after folder discovery, :705), so the exposure window is every run, not just init.
Evidence (verified this run, 2026-10-31)
Code: cold_store.py:373-378 (in-place O_TRUNC write), :381-396 (bare json.loads in both loaders), :398-405 (save_state via the same writer).
Laptop probe (source sha-matched: cold_store.py sha256 fba42e08027514abae2de1602e755854ebfd6a3b578a5eece34b6c0becd39b95 on both sandbox main ed3a285e and the laptop copy; probe dir ~/git/fleet/probe-2026-10-31-laneB-coldstore-atomic/):
- Control:
save_state({"folders": {"a": "1"}})thenload_state()returns the dict. - Simulated crash mid-write (partial JSON flushed, then the write aborts):
state.jsonis left asb'{"folders": {"a": "1"'. - After that,
load_state()raisesJSONDecodeError: Expecting ',' delimiter: line 1 column 22— andisinstance(e, ColdStoreError)isFalse. - Same simulation against
_write_private_json(credentials_path, …):load_credentials()raises the same rawJSONDecodeError.
Contrast within the same repo: pkgs/id's JSONL stores treat a crash-torn tail as a normal, repairable case (e.g. Petnames::open drops and truncates a torn final line). Cold-store's two most state-critical JSON files get no such protection.
Why it matters
Cold-store is the backup system: its state.json carries the Drive folder map every backup/restore run depends on, and credentials.json is hand-filled from Bitwarden — a torn credentials file destroyed by a crash during an unrelated save_state-adjacent write forces manual credential re-entry. A backup tool whose own state files are the least crash-safe part of the system will eventually wedge a nightly run with an inscrutable stdlib traceback instead of an actionable error.
Distinct from
- github-backup(state): plan.json, state.json and MANIFEST.json are written non-atomically and loaded with bare json.load — one torn write wedges the daily fetch, the drainer and manifest regeneration #3240 (github-backup's plan/state/MANIFEST non-atomic writes — different tool, different files).
- cold-store-reaper: prune_backups apply is not crash-consistent — a mid-run trash failure rolls back every ledger row, but the Drive trashes stay, and the trashed files are invisible to any retry #2550 (prune_backups Drive/ledger crash-consistency — remote side, not the local JSON writers).
- cold-store-reaper: live Google access token is cached in state.json, which the README calls '(no secrets)' and which is written without 0600 permissions #940 (closed; token cached in state.json + file permissions — not write atomicity).
Acceptance criteria
-
_write_private_jsonwrites to a temp file in the same directory, fsyncs, andos.replaces onto the destination (preserving 0600), so a crash at any point leaves either the old or the new complete document. -
load_state/load_credentialson a corrupt file raiseColdStoreErrornaming the file and the parse problem (or recover to a documented default), never a rawJSONDecodeError. - Regression test: simulate a mid-write failure (partial write + raise) and assert the previous complete document still loads, for both state and credentials paths.
- Existing cold-store test suite passes.
Activity
dezren39 commentedon Nov 1, 2026
Filed this run by new-issue-creator-code-b (Lane B). Found via: code read of pkgs/cold-store-reaper/cold_store.py (_write_private_json :373-378 in-place O_TRUNC write; bare json.loads in load_state :393-396 and load_credentials :381-391) plus a laptop probe on sha-matched source (cold_store.py sha256 fba42e08027514abae2de1602e755854ebfd6a3b578a5eece34b6c0becd39b95; probe ~/git/fleet/probe-2026-10-31-laneB-coldstore-atomic/): a simulated mid-write crash leaves state.json as partial JSON and load_state/load_credentials then raise raw JSONDecodeError (not ColdStoreError). Deduped against open+closed cold-store issues (#3240 is github-backup's files; #2550 is Drive/ledger crash-consistency; #940 was state contents/permissions). Enqueued as a triage packet on repo-health-code-pilot.
29 remaining items
dezren39 commentedon Nov 2, 2026
agent-note:v1 — picked up
Agent: repo-worker-code (OpenChamber scheduled lane)
Task: repo-health-code-pilot task 5706 (small_fix)
At: 2026-11-02T14:07:25Z
State: claimed; fresh verification complete
Found via: atomic claim from repo-health-code-pilot.
Fresh state: #3279 OPEN, assigned to dezren39; canonical queue PR #4361 OPEN (non-draft, head d64a9e9), checks python/worker pending; no held/user-owned/do-not-merge/do-not-auto-edit restrictions; no competing PR.
Next: resume branch queue/5706-3279-cold-store-atomic in /home/user/git/fleet/5706-3279-cold-store-atomic; re-verify tests; poll CI up to 5 min, park if still pending.
Links: #3279 · #4361 · task 5706
dezren39 commentedon Nov 2, 2026
agent-note:v1 — parked
Agent: repo-worker-code (OpenChamber scheduled lane)
Task: repo-health-code-pilot task 5706
At: 2026-11-02T14:14:42Z
State: parked
Fresh evidence: #3279 OPEN; canonical queue PR #4361 OPEN non-draft at d64a9e9; worktree suite green (120 tests OK, 1 skipped); CI poll 14:09–14:14Z still queued (python unittest suites + worker local proof, run 41448228868).
Disposition: parked for 20 minutes for CI. work/assigned remains set.
Resume: revalidate PR/issue/checks, disposition findings, merge if green, then complete fixed_merged.
Links: #3279 · #4361 · task 5706
dezren39 commentedon Nov 2, 2026
🤖 Agent note · state: picked up
dezren39 commentedon Nov 2, 2026
🤖 Agent note · state: parked
Merge pull request #4361 from developing-today/queue/5706-3279-cold-s…
dezren39 commentedon Nov 2, 2026
🤖 Agent note · done