Repository navigation
cold-store-reaper: nightly-backup status step spuriously FAILED when status prints >20 lines — head closes the pipe, SIGPIPE + pipefail fail a healthy run #1430
Description
Bug
scripts/nightly-backup.sh's status step can mark the whole nightly run FAILED even when cold-store status succeeds — a race between head closing the pipe and the producer finishing its writes.
The step (nightly-backup.sh, status block):
if "${COLD_STORE}" status | head -n 20; thenThe script runs with set -uo pipefail, so the pipeline's exit status is the producer's when the producer is the failing element. When status output exceeds 20 lines, head exits after line 20 and closes the read end; a producer still writing gets SIGPIPE (a shell producer exits 141; the real cold-store is Python, which converts SIGPIPE into BrokenPipeError at flush and exits non-zero). Whether the producer finishes first depends on output size and scheduling — so the step flips between ok and FAILED on identical healthy state.
Reproduction (verified this run, main 37418e56)
Using the script's own documented test seams (COLD_STORE_BIN, EXPORT_TURSO_BIN) with a stub cold-store whose status prints 100 lines and always exits 0, and a stub export-turso that succeeds:
- run 1:
step status: ok, exit 0 - run 2:
step status: ok, exit 0 - run 3:
step status: FAILED, script exit 1,FAILED steps: status— every step actually succeeded
The existing tests in tests/test_nightly_backup.py cannot catch this: their stub status prints exactly one line (stub status summary: objects 0 cold), so head never closes early.
Why it matters
The nightly backup is the cold store's core ritual, and PR #1373 just reworked this script specifically so failures stay loud but not contagious. This bug does the opposite: a spurious status failure makes a fully healthy night exit 1 and report FAILED steps: status, training the operator to ignore the run's verdict — and the failure probability grows every time cold-store status output grows past 20 lines (it is described in the script header itself as a summary shown via head, i.e. truncation is expected).
Acceptance criteria
- The status step reports ok whenever
cold-store statusitself exits 0, regardless of how many lines it prints (verify with a stub printing several hundred lines, run repeatedly — e.g. 20 consecutive runs, zero spurious failures). - A genuinely failing
cold-store status(non-zero exit) still marks the step FAILED and the script still exits non-zero at the end. - The printed summary is still truncated for display (~20 lines) — the fix must not dump unbounded status output into the nightly log.
-
tests/test_nightly_backup.pygains a case whose stub status prints more than 20 lines and assertsstep status: ok+ exit 0, alongside the existing isolation cases.
Activity
developing-today commentedon Oct 27, 2026
CODE-517
dezren39 commentedon Oct 27, 2026
Filed this run by the new-issue-creator (lane A). Reproduced on main
37418e56via the script's ownCOLD_STORE_BIN/EXPORT_TURSO_BINtest seams: a stubcold-store statusprinting 100 lines and exiting 0 made the run reportstep status: FAILEDand exit 1 in 1 of 3 runs (head -n 20 closes the pipe; SIGPIPE + pipefail). The existing nightly tests use a one-line stub status, so they cannot see it.dezren39 commentedon Oct 28, 2026
Queue task 1434 progress — fix pushed.
What changed (scope-limited to the status step + regression test):
pkgs/cold-store-reaper/scripts/nightly-backup.sh: status step no longer pipescold-store statusdirectly intohead -n 20underset -o pipefail. It captures full output to a mktemp file, records cold-store's own exit code, thenhead -n 20the file purely for display. Verdict depends only on cold-store's exit code; FAILED line now includes the exit code, matching run_step style.pkgs/cold-store-reaper/tests/test_nightly_backup.py: newtest_long_status_output_still_ok_and_truncatedwith a 300-line stub status asserting exit 0,step status: ok, lines 1–20 shown, lines 21/300 absent.Test results (laptop, repo conventions):
python3 -m unittest pkgs/cold-store-reaper/tests/test_nightly_backup.py -v: 4/4 OKstep status: FAILED (exit 3), script exit 1, display still truncatedPR: #1774 (references #1430, not merged).
dezren39 commentedon Oct 28, 2026
🤖 Dispatcher binding · openchamber-dispatcher run 2026-10-28 13:45 CDT (burn-by-expiry run: Drew directive 13:44 CDT — fill free session slots, burn the Go window resetting 2:26 PM and the Meta window resetting 4:23 PM)
small_fix) ↔ issue cold-store-reaper: nightly-backup status step spuriously FAILED when status prints >20 lines — head closes the pipe, SIGPIPE + pipefail fail a healthy run #1430ses_e7c242effffelJMyBEjTqBEMM6(laptop OpenChamber, agent build)meta/muse-spark-1.3-contributor(Meta pool, Muse Code High Usage plan; fresh quota read this run ~13:47 CDT: 5h 1% / weekly 11%, reset 4:23 PM — OPEN, plan quota only, never PAYG; seated on Meta per the run's burn-by-expiry directive so the Meta 5h window does not expire near-0%)~/git/fleet/1434-1430-nightly-backup-sigpipeon the laptop, branchqueue/1434-1430-nightly-backup-sigpipe(from origin/main 35726e8)queuedin the fleet queue (no targeted claim/binding verb exists yet — Dispatcher v1 live: scheduled launches of OpenCode/OpenChamber sessions for queued issues #1337/Routing policy alignment: map OpenChamber routing categories to the fleet's Workhorse-first tiers + POOL MAP gates #1272); the owning lane reconciles/harvests from here, and the babysitter shepherds the session.dezren39 commentedon Oct 29, 2026
🤖 Agent note · bound