Repository navigation
cold-store-reaper: resumable upload loops forever on a 308 whose Range stops advancing — no progress check or retry cap, and a Range-less 308 counts unacknowledged bytes as sent #2797
Description
Problem
ColdStore._resumable_upload (pkgs/cold-store-reaper/cold_store.py:812-843, main c90fcbc7) loops on HTTP 308 with no progress check, no retry cap, and no stall detection:
while sent < total or total == 0:
...
if status == 308:
range_hdr = headers.get("Range") or headers.get("range") or ""
try:
sent = int(range_hdr.rsplit("-", 1)[1]) + 1
except (IndexError, ValueError):
sent += len(chunk)
f.seek(sent)
continueTwo verified failure modes in the same loop:
- A 308 whose
Rangestops advancing loops forever.sentis recomputed from the server's acknowledged range on every iteration, so if Drive keeps answering308 Range: bytes=0-0(session wedged, bytes landing nowhere),sentis pinned at 1 and the client re-PUTs the same chunk indefinitely — no maximum attempts, no requirement thatsentincrease, no backoff. The upload never fails, never finishes, and the nightly job hangs while hammering the Drive API. - A 308 with no
Rangeheader is treated as if the whole chunk landed. The fallbacksent += len(chunk)assumes bytes the server never acknowledged (per the Drive resumable protocol, a 308 withoutRangemeans zero bytes received so far). For a single-chunk file the client then exits the loop and reportsupload finished without a file id— a misleading error that hides the actual incomplete upload; for a multi-chunk file it skips unacknowledged bytes and only the post-upload md5 check inupload()catches the resulting corruption, after the whole file has been sent.
The existing fake-Drive test (tests/test_cold_store.py:186-212) always returns a Range that advances by exactly the bytes received, so neither case is exercised.
Evidence (verified this run, main c90fcbc7)
Laptop probe (~/git/fleet/probe-2026-10-30-laneB-coldstore308/, real module imported from the laptop clone; cold_store.py md5 882a83a1d80b409192d5ae1db3ccb975 matches the sandbox clone). Transport monkeypatched; no credentials involved:
probe.py (PUT always -> 308, Range: bytes=0-0; sentinel cap at 30 PUTs)
RESULT: PROBE-CAP: 30 PUTs reached, loop still going
PUTs issued: 31 (file is 1000 bytes, chunk cap 8MiB -> a healthy upload needs 1-2 PUTs)
probe2.py (PUT always -> 308, no Range header)
RESULT: ColdStoreError: upload finished without a file id
PUTs issued: 1 (server acknowledged 0 bytes, yet the client treated the chunk as sent and left the loop)
Dedup: title + keyword search over open and closed issues (cold-store resumable, 308, upload loop) finds nothing covering this loop — #2733 is the S3 list-page parser in the same package, #938 (closed) was upload-limit enforcement in upload()/put_backup.
Why it matters
cold-store-reaper is the fleet's backup path to Drive; its uploads run unattended (nightly). A wedged resumable session currently turns into an infinite, silent PUT loop instead of a failure the job can report and retry tomorrow — the bundle stays in uploading state and the run never completes. The missing-Range fallback additionally misreports an incomplete upload as a finished one.
Acceptance criteria
-
_resumable_uploadterminates with aColdStoreErrorwhen a 308 response does not advance the acknowledged offset (no-progress detection), and/or after a bounded number of 308 retries — a stuck session fails the upload instead of looping forever. - A 308 with a missing or unparseable
Rangeheader no longer assumes the whole chunk was received; the client re-sends from the last server-acknowledged offset (or queries the session status, per the Drive resumable protocol) — triage pins the exact approach. - Tests extend the fake-Drive coverage with (a) a 308 whose
Rangenever advances and (b) a 308 with noRangeheader; the existing resumable-upload tests still pass. - Healthy-path behavior is unchanged: advancing 308s resume from the acknowledged byte, and a completed upload still returns the file id.
Activity
dezren39 commentedon Oct 30, 2026
🤖 Agent note · filed
2 remaining items
dezren39 commentedon Nov 1, 2026
Correction to the findings comment above: the follow-up small_fix is queue task 5413 (the placeholder in that comment did not render the id).
🤖 Agent note · done
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done