Repository navigation
cold-store-reaper: Drive list_backups never follows nextPageToken — prune-backups retention plans from a silently truncated listing #2426
Description
cold-store-reaper: Drive list_backups never follows nextPageToken — retention plans from a silently truncated listing
Evidence (verified this run, main 5a34fa7)
ColdStore.list_backups (pkgs/cold-store-reaper/cold_store.py:992-1031) issues exactly one Drive files.list request per db folder with pageSize: 1000 and reads only resp["files"]. It never looks at nextPageToken and never sends a pageToken, so when a db-backups/<db>/ folder holds more than one page of files, every file past the first page is invisible to all consumers.
The listing is ordered createdTime desc (cold_store.py:1011), so the invisible files are the oldest ones — precisely the files retention exists to trash:
prune_backups(cold_store.py:1092+) builds its 30 daily / 12 weekly / 12 monthly plan (_retention_plan,cold_store.py:1048) fromlist_backupsrows only. Files beyond page 1 are never kept, never pruned, and never reported as unparseable — they silently accumulate in Drive forever.- The plan's "distinct days/weeks/months present" computation is likewise computed over a truncated set whenever the folder exceeds one page.
Probe (this run, sandbox python3, real cold_store.py with a stubbed http_request emulating the Drive API — page 1 returns 2 files plus nextPageToken: tok2, page 2 returns a third, older file):
rows returned: 2 ['d1-20261029T010000Z-a.sql', 'd1-20261028T010000Z-a.sql']
requests made: 1
any request carried pageToken: False
The third (oldest, 2025-01-01) backup is dropped with no error.
Probe kept at ~/workspace/goals/keep-the-four-repos-healthy/hidden_files/probe-2026-10-29-laneB-drivepage/probe.py.
Contrast in the same file
The S3 listing path paginates correctly: _parse_s3_list_page returns the continuation token (cold_store.py:138-172) and its caller loops over pages — the test suite even asserts the page sequence ("fake pages are 2, 2, 1", tests/test_cold_store.py:1325). The Drive path has no equivalent loop, and the test fake (FakeDrive) implements no pagination at all, so the gap is untestable today.
Why it matters
Retention is the only mechanism that bounds Drive growth for DB backups, and it runs nightly (scripts/nightly-backup.sh step 4: prune-backups --apply). Parseable backups are capped by retention itself (~54 per db), but unparseable names are never pruned by design (reported only), pre-retention/manual files count too, and any folder that crosses 1000 files permanently hides its oldest members from the one tool meant to remove them. The failure is silent: the nightly run reports success while Drive grows.
Related but distinct: #1632 is the same class of bug in the S3 listing loop (continuation-token handling); this issue is the Drive files.list path, which never paginates at all.
Acceptance criteria
list_backupsfollowsnextPageTokenuntil the listing is exhausted, preserving newest-first order across pages.- A folder whose listing spans multiple pages yields the full row set;
prune_backupsplans over the complete set (an over-page oldest excess file appears in the prune plan in a test). - The
FakeDrivetest double implements Drive pagination (honourspageSize/ emitsnextPageToken), with a regression test that a multi-page folder is fully listed. - Existing cold-store test suites still pass.
Activity
developing-today commentedon Oct 29, 2026
CODE-1238
dezren39 commentedon Oct 29, 2026
🤖 Agent note · filed
13 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · state: picked up
ColdStore.list_backups, with FakeDrive pagination and regression tests.nextPageToken; avoid touching other Drive or S3 listing paths.dezren39 commentedon Nov 1, 2026
CI resume
PR #3431 is open, but both
python: unittest suitesandworker: local proofare still queued. The task is parked for 20 minutes; resume by re-reading PR checks and proceed only when they settle. The branch isqueue/5320-2426-drive-list-backupsatad55d9803; local full tests passed (88 passed, 1 skipped).🤖 Agent note · state: parked
queue/5320-2426-drive-list-backups