Repository navigation
Description
Summary
cold-store-reaper's S3 ListObjectsV2 parser (_parse_s3_list_page) trusts the response body completely: it never validates that the XML root is a list result, never converts XML parse failures to the module's ColdStoreError, parses Size with a bare int(...), and accepts negative sizes. A malformed or shape-drifted R2/proxy response therefore either crashes the CLI with a raw traceback, or — worse — is silently read as an empty/negative-sized listing that downstream backup and eviction code trusts.
Evidence (main ba5229e9, pkgs/cold-store-reaper/cold_store.py)
_parse_s3_list_page(:138) runsET.fromstring(body)at :146 with no guard.main()catches onlyColdStoreError(end ofmain, ~:2240), so a non-XML body escapes as a rawxml.etree.ElementTree.ParseErrortraceback.Sizeis parsed at :176 asint(fields.get("Size") or 0)with no guard:"abc"/"1.5"raise a rawValueError;"-5"is accepted and stored as a negative object size.- The root element is never checked: any well-formed XML that is not a
ListBucketResult(e.g. an HTML/XML error or maintenance page returned with HTTP 200 —_s3_checkat :1557 only rejects status >= 300) yields([], False, None), i.e. "the bucket is empty", froms3_list_objects(:1563, parse call :1574). - The parsed sizes are load-bearing downstream:
backup_supabasepassesnewest["size"]as the byte-count verification for a downloaded backup (:1789), andevict_applywritesentry["size"]into the transfer ledger (:1915);evict_scan(:1817) plans deletions off this same listing.
Verified this run with the real module (md5-matched copy of main's cold_store.py; probe on the laptop at ~/git/fleet/probe-2026-10-30-laneB-s3parse/probe.py, stdlib-only):
- control: valid
ListBucketResultpage →[{'key': 'a.bin', 'size': 3, 'last_modified': 2026-10-01T00:00Z}], truncated False (OK) - body
{"error": "nope"}(non-XML) → RAWParseError: not well-formed (invalid token) <Size>abc</Size>/<Size>1.5</Size>→ RAWValueError: invalid literal for int()<Size>-5</Size>→ accepted, object size-5- well-formed
<html><body>proxy error</body></html>→ silently[], truncated False (read as an empty bucket)
Why it matters
This listing is the input to backup-supabase and to eviction, which deletes objects. The module's own contract elsewhere is fail-closed with named ColdStoreErrors (transport retries, the #2110 aware-or-None datetime rule, the curl size verification). The list parser is the one place a drifted body can either mask the real failure behind an unrelated traceback or, in the wrong-root case, present a non-empty bucket as empty to code that plans from it.
Acceptance criteria
_parse_s3_list_pageraisesColdStoreError(naming the problem) for a body that is not well-formed XML and for a well-formed document whose root is not a list result.- A
Contentsentry whoseSizeis missing keeps the current default; aSizethat is non-numeric or negative raisesColdStoreErrorinstead of a rawValueErroror a negative object size. - Regression tests cover: non-XML body, wrong-root XML body,
Size=abc,1.5,-5, plus the valid-page control (including the existing pagination and LastModified behaviour). - Existing cold-store-reaper tests pass unchanged.
Dedup
- cold-store-reaper: D1 response handling assumes object shapes throughout — array/string bodies and non-object errors/result entries raise AttributeError instead of ColdStoreError #2618 covers the D1 response shapes in this same file and its queue packet explicitly scoped S3 response handling (
s3_error_message/_parse_s3_list_page) OUT — this filing is that acknowledged remainder, a different subsystem (R2/S3 listing vs D1 API) with different failure modes. - cold-store-reaper: s3_list_objects loops forever on a truncated page with a missing, empty, or repeated continuation token #1632 covers the
s3_list_objectspagination loop (missing/repeated continuation token); cold-store-reaper: an offset-less LastModified in an S3 listing parses to a naive datetime and crashes backup_supabase / evict_scan — _parse_s3_list_page promises aware-or-None #2110 (closed) covered offset-lessLastModifiedparsing in this same function — neither touches root validation, XML parse failures, orSizeparsing. - Searched open + closed issues for S3 parse / malformed XML this run: no match.
Activity
dezren39 commentedon Oct 30, 2026
🤖 Agent note · filed
23 remaining items
dezren39 commentedon Oct 31, 2026
agent-note:v1
event: parked
task: 4414 (lane F)
artifact: PR #3133 @ 95bf268
state: Codex clean on head (re-verified). Dispatched nix-ci run 41224029839 on the branch — still QUEUED after 5-min poll (fleet backlog). Laptop suite 101 tests OK per opening lane.
resume: run green -> merge -> #2733 auto-closes -> complete fixed_merged.
dezren39 commentedon Oct 31, 2026
🤖 Agent note · picked up
fix(cold-store-reaper): fail closed on malformed S3 list pages (#2733) (
dezren39 commentedon Oct 31, 2026
🤖 Agent note · done