Repository navigation
cold-store-reaper: an offset-less LastModified in an S3 listing parses to a naive datetime and crashes backup_supabase / evict_scan — _parse_s3_list_page promises aware-or-None #2110
Description
cold-store-reaper: an offset-less LastModified in an S3 listing parses to a naive datetime and crashes backup_supabase / evict_scan
Evidence (main 82b019a0)
pkgs/cold-store-reaper/cold_store.py:138-165 — _parse_s3_list_page documents its own contract:
Each object is {key, size, last_modified}; last_modified is an aware datetime, or None when absent/unparseable — callers treat None as oldest, never as 'now'.
The parse at :162 breaks that contract:
stamp = datetime.fromisoformat(raw.replace("Z", "+00:00"))An ISO-8601 stamp with no timezone designator (e.g. 2026-10-01T00:00:00.000 — a form S3-compatible stores emit; this module already anticipates non-AWS reply variation, cf. the namespace note in the same docstring) parses successfully to a naive datetime — not aware, not None. Every downstream consumer assumes aware:
cold_store.py:1780(backup_supabase):max(objects, key=lambda o: (o["last_modified"] or epoch, o["key"]))withepochaware — comparing a naive stamp against the aware epoch (or another object's aware stamp) raisesTypeError: can't compare offset-naive and offset-aware datetimes.cold_store.py:1837(evict_scan): the same tuple sort over the whole bucket.cold_store.py:1861(evict_scan):now - obj["last_modified"]withnowaware raisesTypeError: can't subtract offset-naive and offset-aware datetimesfor any naive-stamped object that reaches the age check.
Reproduction (verified this run, shipped module imported unmodified)
_parse_s3_list_page(<Contents>…<LastModified>2026-10-01T00:00:00.000</LastModified>…)
-> last_modified = 2026-10-01 00:00:00, tzinfo = None # contract says aware-or-None
datetime.now(timezone.utc) - stamp
-> TypeError: can't subtract offset-naive and offset-aware datetimes
sorted(objects, key=lambda o: (o["last_modified"] or aware_epoch, o["key"]))
-> TypeError: can't compare offset-naive and offset-aware datetimes
Why it matters
One offset-less stamp anywhere in a bucket listing takes down the whole backup_supabase run and the whole eviction scan with an unhandled TypeError — no object is filed, no verdict is produced, and the traceback names datetime internals rather than the offending key. The configured R2 backend currently emits Z stamps, so this is latent until the tool is pointed at (or proxied/stubbed by) a store that omits the offset — exactly the variation this parser already defends against elsewhere. The fix direction is triage's pin: treat an offset-less stamp as unparseable (→ None, the documented safe meaning: oldest / never age-eligible) or interpret it as UTC — the two differ in eviction age verdicts, so the choice should be explicit.
Acceptance criteria
-
_parse_s3_list_pageupholds its documented contract for every input:last_modifiedis always an aware datetime orNone(offset-less stamps handled by the pinned rule, with a comment naming it). -
backup_supabaseandevict_scancomplete withoutTypeErroron a listing mixing aware, offset-less, and missingLastModifiedvalues. - Tests cover: offset-less stamp page; a mixed listing through
evict_scan(the offset-less object gets a non-evictable verdict, never a crash) and through thebackup_supabasenewest pick. - Existing behaviour unchanged for
Z/ explicit-offset stamps and for absentLastModified.
Dedup
Searched open + closed issues for cold-store, s3_list, LastModified, naive, evict_scan: #1632 (open) is the continuation-token loop in the same listing function — a different defect with a different fix; #1427/#1428/#1430/#1435 concern CI, shell plumbing, env validation and test speed. No issue covers offset-less timestamp parsing.
Activity
developing-today commentedon Oct 29, 2026
CODE-1015
dezren39 commentedon Oct 29, 2026
🤖 Agent note · filed
dezren39 commentedon Oct 29, 2026
🤖 Agent note · TRIAGE VERDICT: CONFIRMED — reproduced on
main5a34fa70(openchamber scheduled lane, 2026-10-29T21:54Z)Independently reproduced against the shipped module imported unmodified from a clean worktree at
origin/main. The issue is accurate, and the reproduction adds one refinement about which consumer crashes.Confirmed mechanism
pkgs/cold-store-reaper/cold_store.py:162—datetime.fromisoformat(raw.replace("Z", "+00:00")). An offset-less stamp parses successfully to a naive datetime, so theexcept ValueError: stamp = Nonesafety net never fires and the function's own docstring contract ("aware datetime, or None") is broken at that line and nowhere else.Reproduction (verbatim output)
Refinement the issue states too broadly — worth pinning in the fix
The two failure modes have different trigger conditions, and the fix's tests should cover both:
sortedinevict_scan,maxinbackup_supabase) crashes only when a naive stamp is compared against an aware one — i.e. a mixed listing, or a listing where a neighbour is absent (absent →None or epoch→ aware). A listing where every stamp is offset-less sorts fine (case 5). So "one offset-less stamp takes down the scan" is true only when the listing is not uniformly offset-less.now - obj["last_modified"],cold_store.py:1861) crashes on any single naive stamp that reaches the age check, with no neighbour required (case 5). This is the unconditional crash: one offset-less, hash-known, manifest-coldobject is enough to kill the whole eviction scan.backup_supabaseis therefore the narrower exposure (mixed listing only) andevict_scanthe wider one.Pinned rule (the choice the issue left open)
Treat an offset-less stamp as unparseable →
None. Reasons:Noneyieldstoo-young, which is non-evictable by construction.Implementation is a one-line guard plus a comment naming the rule, e.g.
if stamp.tzinfo is None: stamp = None # offset-less = zone unknown = unparseable (contract: aware-or-None)— after the existingtry/except ValueError.Test matrix the fix should carry
None(contract test, currently the failing one)evict_scan→ completes, offset-less objecttoo-young, neverTypeErrorbackup_supabase's newest pick → completes, newest is an aware-stamped objectZ, explicit-offset, and absentLastModifiedScope note
Triage only — this lane changed nothing: no branch, no PR. The worktree used for the repro has been removed. #1632 (continuation-token loop, same function) remains a separate defect; the fix here should not touch the token path.
pkgs/cold-store-reaper/cold_store.py:162,1780,1837,1861· cold-store-reaper: s3_list_objects loops forever on a truncated page with a missing, empty, or repeated continuation token #1632dezren39 commentedon Oct 30, 2026
🤖 Agent note · picked up
_parse_s3_list_pagereturns None for an offset-less LastModified (never a naive datetime, never reinterpreted as UTC), with offset-less + mixed + uniformly-offset-less tests inpkgs/cold-store-reaper/tests/test_cold_store.py. Branch + PR; tests on the laptop/CI per fleet policy.Merge pull request #2725 from developing-today/queue/3656-2110-offset…
dezren39 commentedon Oct 30, 2026
🤖 Agent note · done
1c95280d):_parse_s3_list_pagenow maps an offset-lessLastModifiedtoNoneper triage 3066's pinned rule (zone unknown = unparseable; never reinterpreted as UTC), with the full test matrix from triage (parser contract, mixed-listingevict_scan, uniformly-offset-lessevict_scan,backup_supabasenewest pick). Full test file on the laptop: 78 tests OK (skipped=1). Codex review on the PR: completed, no findings. CI note: the only red runs on the branch were thejules-daily-*workflows failing with zero jobs — verified to fail identically on every queue branch push today (pre-existing infra, unrelated to this change); filed as ci: jules-daily workflows fail with zero jobs on every branch push #2728.