cold-store: hash/compress two-pass race seals objects whose manifest hash never matches their bytes — silently unrestorable backups (Codex P1 on #923, merged unacted)

Defect

The cold-store reaper can seal an archive that can never be restored: hashing and compression read the source in two separate passes, so a file that changes between them is recorded in the manifest with the old hash but compressed with the new bytes. Upload verification passes and the object is marked cold, but every future fetch rejects the restored bytes with a SHA-256 mismatch. Codex flagged this as a P1 on PR #923; the PR merged without the finding being acted on. Defect confirmed present on main 2026-10-29 (audit: REAL + never engaged + merged anyway).

The Codex finding (verbatim, PR #923)

Hash and compress the same stable source snapshot

If a source file changes between sha256_file(src) and the separate read performed by zstd, the manifest records the old content hash while the bundle contains the new content. The subsequent upload verification still succeeds and marks that object cold, but fetch will always reject its restored bytes with a SHA-256 mismatch. This can silently create an unusable archive for actively written files. Hash and compress a stable snapshot, or verify that the compressed frame decodes to the recorded object hash before sealing.

Finding: #923 (comment) (filed against pkgs/cold-store-reaper/cold_store.py line 392, PR #923)

Fix direction (as Codex prescribed)

Hash and compress a single stable snapshot of the source, or verify that the compressed frame decodes to the recorded object hash before sealing the object as cold.

Source

Codex review-value audit 2026-10-29 (research_notes/codex-review-value-20261029/findings.md §2, best catch #3): 636 Codex findings classified, this one REAL-CATCH, never engaged, PR merged anyway. Verification: defect confirmed present on main 2026-10-29.

Acceptance criteria

Hash and compression consume one stable snapshot of the source (or the compressed frame is verified to decode to the recorded hash before sealing) — the two-pass race in the body can no longer seal a mismatched object
A regression test mutates the source between the old pass points (or simulates it) and shows the object is either sealed consistently or refused, never sealed cold with a hash its bytes fail
Objects already sealed mismatched, if any are identifiable, are reported on this issue rather than silently left

CODE-1132

Intake review

Related: PR #923 (Codex P1 finding r4372844010, quoted in the body; merged unactioned), pkgs/cold-store-reaper/cold_store.py line ~392

Notes: Filed from the Codex P1 on merged PR #923 and confirmed present on main 2026-10-29: sha256_file and the zstd read are separate passes over a live file, so a mid-seal change records the old hash over new bytes; upload verification passes, the object is marked cold, and every future fetch rejects it — a silently unrecoverable archive. Already labelled kind/bug, priority/high, effort/s; this pass appends acceptance criteria and marks enriched.

Risks: Cold storage is the last copy by design; an object that can never be fetched is data loss discovered at restore time, the worst moment to learn it.

Suggested approach: Snapshot the source once (temp copy or fd-stable read) and hash the exact bytes compressed; add the pre-seal decode-verify as belt-and-braces; regression-test with a mutating writer.

🤖 Agent note · filed

Agent: fleet monitor (repo-fleet-monitor)
Session / ID: repo-fleet-monitor run 2026-10-30T02:12Z (2026-10-29 21:12 CDT)
Found via: monitor reconciliation
Task: Duty 2 intake enrichment — labels normalized and meta/enriched applied; acceptance criteria appended to the body only where no acceptance section existed (original body preserved verbatim above it). Still-live check this pass: no merged PR or fix commit explicitly resolving this issue was found in this pass's search.
Expectation: see the effort label; done = the acceptance criteria are met.
Links: this issue

agent-note:v1
task: 3424
state: handed_off
lane: code-j
execution_lane: opencode
note: Picked up by Lane J shift 2026-11-03; execution-lane assignment returned opencode, so task 3424 is handed off to the opencode claimant, which launches a bound session. No direct work by this lane.

🤖 Agent note · handed off

Agent: fleet monitor
Session / ID: repo-fleet-monitor run 2026-11-02 20:12 CST
Found via: monitor reconciliation
Task: Claim-evaporated repair: task 3424 is queued unclaimed (attempts 3/3, checkpoint stage handed_off) while this issue still carried claimed labels. Labels reset to queue/queued + status/needs-triage so GitHub matches the queue. The task is DEAD at its attempt cap and can never be claimed again — operator repair: raise the cap or enqueue a fresh packet resuming from stage handed_off.
Expectation: operator/lane pickup after repair; no holder is currently working this issue.

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-g
Session / ID: task 3424
Found via: queue claim
Task: task 3424 (fix for issue cold-store: hash/compress two-pass race seals objects whose manifest hash never matches their bytes — silently unrestorable backups (Codex P1 on #923, merged unacted) #2269 — cold-store hash/compress two-pass race seals unrestorable backups) is assigned to the opencode execution lane; handing off per worker instruction §1b — this lane does not work it directly and creates no worktree for it.
Expectation: the opencode claimant (polls every 10 minutes for claimed, opencode-assigned, unbound tasks) launches a bound session that drives task 3424 to its terminal state. If no session launches before the claim lease expires, task 3424 returns to the queue.
Links: issue cold-store: hash/compress two-pass race seals objects whose manifest hash never matches their bytes — silently unrestorable backups (Codex P1 on #923, merged unacted) #2269

🤖 Agent note · state: handed off

Agent: opencode-claimant
Task: 3424 (repo-health-code-pilot)
Session / ID: ses_e5e5f82dce0ffe9ZwKoaca2Hct
Route: opencode-route/workhorse, agent build
Claiming worker: repo-worker-code
Status: session created, bound in the queue lane ledger/task row, checkpointed as launched, and task packet delivered.

fix(cold-store): verify frame decodes to recorded hash before sealing (…

🤖 Agent note · done — orphaned fix verified and landed

Agent: queue worker (repo-worker-code), lane bf419306-c4ef-470f-a553-beabf93239de (openchamber scheduled lane)
Session / ID: task 7450 (coverage-pass triage)
Finding: work existed but its owning task had evaporated. PR Fix #2269: verify cold-store frame decodes to recorded hash before sealing #4800 (fix-2269-cold-store-pack-race, head b800eb0b, opened 2026-11-03T08:22Z by the opencode session bound to task 3424) sat open, clean and merge-ready, with no live queue task behind it — the exact silent-abandonment pattern the coverage pass was created to catch.
Verified on the PR head before merging:
- python3 -m unittest discover -s tests in pkgs/cold-store-reaper → 126 tests OK, 1 pre-existing skip (matches the PR's stated verification).
- Diff matches intent precisely: pack() streams each frame through zstd -d, hashing the decoded bytes, and refuses with ColdStoreError when they do not match the recorded object hash; the sealed member size becomes the verified decoded size (no third racy stat()); a failed pack removes bundle.tmp/frame.tmp so nothing half-written lingers. Regression test mutates the source between the hash and compress passes and asserts refusal plus zero rows/artifacts.
- mergeable_state=clean; basic production-line-limit check SUCCESS; CodeRabbit NEUTRAL; the Codex "failed" review state is only its usage-limit notice (no findings body, no inline comments).
- No held / user/owned / do-not-merge / do-not-auto-edit labels.
Outcome: PR Fix #2269: verify cold-store frame decodes to recorded hash before sealing #4800 merged as 927ecc96d5; issue closed as fixed. Labels work/assigned, queue/claimed, status/in-progress cleared.
Resume note for task 3424's owner: nothing left to do — the fix is on main.

Navigation Menu

Repository navigation

cold-store: hash/compress two-pass race seals objects whose manifest hash never matches their bytes — silently unrestorable backups (Codex P1 on #923, merged unacted) #2269

Defect

The Codex finding (verbatim, PR #923)

Fix direction (as Codex prescribed)

Source

Acceptance criteria

Intake review

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cold-store: hash/compress two-pass race seals objects whose manifest hash never matches their bytes — silently unrestorable backups (Codex P1 on #923, merged unacted) #2269

Description

Defect

The Codex finding (verbatim, PR #923)

Fix direction (as Codex prescribed)

Source

Acceptance criteria

Activity

developing-today commented on Oct 29, 2026

dezren39 commented on Oct 30, 2026

Intake review

dezren39 commented on Nov 3, 2026

4 remaining items

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions