Repository navigation
cold-store-reaper: s3_list_objects loops forever on a truncated page with a missing, empty, or repeated continuation token #1632
Description
Summary
ColdStore.s3_list_objects (pkgs/cold-store-reaper/cold_store.py:1555-1569) paginates ListObjectsV2 with no progress guard: it loops while True, extending the result list, and returns only when a page says IsTruncated is false. If a page claims truncation but carries no usable continuation token — or repeats the token just used — the loop re-requests the same page forever, appending duplicate objects and growing memory without bound. There is no error, no page cap, and no token-progress check.
Evidence (verified this run, main ed8e5910)
s3_list_objects(cold_store.py:1559-1569):token = None; each iteration buildsquery = {"list-type": "2"}and addscontinuation-tokenonlyif token:(:1561-1563 — an empty-string token is dropped too), calls_s3_request, extendsout, and returns onlyif not truncated._parse_s3_list_page(cold_store.py:138-172) returnstoken = Nonewhen theNextContinuationTokenelement is absent (:149, :154-155) andchild.textverbatim otherwise — it never cross-checks the token against truncation.- Probe on the laptop driving the real module (cold_store.py sha256
d3fe48c5identical in both clones;_s3_requeststubbed to serve canned ListObjectsV2 pages; probe + log in the fleet'shidden_files/probe-2026-10-28-laneB-s3list/):- CONTROL — page 1 truncated with token
tok2, page 2 not truncated: terminates, 2 calls, 3 objects. - BUG-A — every page
IsTruncated=true, noNextContinuationTokenelement: still looping past the probe's 12-call cap (the first page is re-requested with no token each time). - BUG-B — every page truncated and repeats the same token
tokX: still looping past the cap (each iteration re-requests the identical page and appends its objects again). - BUG-C — truncated with an empty token element: still looping past the cap (
if token:drops it, same as BUG-A). - Probe run twice, identical results.
- CONTROL — page 1 truncated with token
- Existing tests cover only well-behaved pagination: the FakeS3 server in tests/test_cold_store.py always emits a fresh
NextContinuationTokenwhen truncated (:760-785), andtest_list_pagination(:1318-1332) asserts the 2/2/1 happy path. No test serves a truncated page with a missing, empty, or repeated token. - Callers inherit the hang:
backup_supabase(cold_store.py:1775) and the eviction scan (:1832) both start froms3_list_objects, so one malformed listing page wedges the nightly backup / evict-scan with no diagnostic — the same class of silent first-request death this package has been hardening against (cf. the transport-retry and curl-fallback work in cold-store-reaper: curl fallback for GET-class S3 requests (LIST) after urllib exhaustion #1625).
Real S3/R2 normally sends a token with IsTruncated=true, but this client already treats the endpoint as untrusted at every other layer (byte-count verification, transport retries, torn-body handling); a non-conforming, proxied, or partially corrupted listing response should be a loud ColdStoreError, not an infinite loop with unbounded memory growth in a long-lived nightly job.
Why it matters
backup-supabase and evict-scan are unattended nightly paths. A single bad listing page turns them into a silent infinite loop: the job never completes, never errors, and accumulates duplicate object entries in memory until it is killed — indistinguishable from a slow run until someone notices the backup is stale.
Acceptance criteria
-
s3_list_objectsraisesColdStoreError(naming the bucket) when a truncated page arrives with a missing or emptyNextContinuationToken. -
s3_list_objectsraisesColdStoreErrorwhen the returned continuation token does not advance (equals the token used for the current request), instead of re-requesting the same page. - Well-behaved multi-page listings behave exactly as today (existing
test_list_paginationand the S3 transport-retry tests pass unchanged). - New tests cover the missing-token, empty-token, and repeated-token pages and assert the call count stays bounded (the listing aborts on the offending page, it does not keep fetching).
Activity
developing-today commentedon Oct 28, 2026
CODE-652
dezren39 commentedon Oct 28, 2026
🤖 Agent note · filed
1 remaining item
dezren39 commentedon Oct 31, 2026
🤖 Agent note · triaged — CONFIRMED
b4136991. Fresh read ofpkgs/cold-store-reaper/cold_store.py:1563:s3_list_objectsloopswhile Trueand returns only when a page reports not-truncated; the continuation token is sent onlyif token:, so a truncated page with a missing/empty token re-requests page 1 and a repeated token re-requests the same page — forever, appending duplicates. No progress guard exists. The packet's probe evidence (mained8e5910, run twice: control terminates in 2 calls; missing/empty/repeated-token pages all loop past the 12-call cap) stands unrebutted, and no open PR addresses this loop (issue cold-store-reaper: S3 list-page parser trusts the body — non-XML/wrong-root pages and bad Size values crash raw or read as an empty bucket #2733 is the separate list-page parser-trust defect).dezren39 commentedon Oct 31, 2026
🤖 Agent note · done
triagedb4136991; implementation follow-up enqueued as queue task 4065. This issue stays open for the fix.dezren39 commentedon Oct 31, 2026
🤖 Agent note · picked up
fix(cold-store-reaper): abort s3_list_objects on a non-advancing cont…
dezren39 commentedon Oct 31, 2026
🤖 Agent note · done