Repository navigation
pm-stack(turso): mirror storage surging +0.48 GB/h — 86.9% of the 5 GB cap at 2026-11-04T13:15Z, binds today at pace #5468
Description
Filed by the pool-quota steward (149th run, 2026-11-04T13:30Z) from the quota gatherer's Turso leg — the surge flagged in the steward ledger at the 147th and 148th runs has accelerated and now has a filing of its own, per the file-the-issue rule.
Measured series (hub quota-current.json, source openchamber-quota, Turso leg)
| Run | Tick (UTC) | Storage | Rows written | Rows read / mo |
|---|---|---|---|---|
| 147th | 2026-11-04T11:15Z | 3.463348 GB (69.3%) | 148,184 | — |
| 148th | 2026-11-04T12:15Z | 3.866595 GB (77.332%) | 191,940 (+43,756) | 411,185,113 (82.237%) |
| 149th | 2026-11-04T13:15Z | 4.342985 GB (86.860%) | 247,467 (+55,527) | 413,260,794 (82.652%, +2,075,681) |
- Storage grew +0.476 GB in the last hour (vs +0.403 GB the hour before). At the current pace the 5 GB free-tier storage cap binds in ~1.4 hours; even at half pace it binds today, 2026-11-04.
- Rows-read is at 82.652% of the 500M monthly cap with the cycle resetting Dec 1 — it binds later than storage at current pace, but it is the second wall in line.
- Databases 3/100 — not the constraint. The growth is in one mirrored dataset's storage + write volume, not database count.
What is needed
- Identify the writer(s) driving ~55k rows-written/hour and ~0.48 GB/hour of storage growth in the Turso mirror (which table(s), which sync/producer).
- Decide the remedy: prune/compact the mirror, slow or scope the sync, or raise the Turso plan — a plan change is Drew's call (money); the steward does not change plans, funds, or credentials.
- Until then the steward keeps reporting the Turso leg hourly in the pool-change ledger; the board attention items already surface "Turso — Storage 87%" and "Turso — Rows read / month 83%".
Acceptance
- The growth source is named (table + producer), and either the growth rate drops back inside the free-tier caps with headroom to the Dec 1 cycle reset, or a plan decision is recorded here.
Activity
developing-today commentedon Nov 4, 2026
CODE-1329
dezren39 commentedon Nov 4, 2026
Steward 150th run (2026-11-04T14:32Z) — ESCALATION, threshold crossed: Turso storage is now 4.774683 GB / 5 GB = 95.494% (was 4.342985 GB = 86.860% at the 149th run, +0.4317 GB in one hour; at this pace the 5 GB cap binds in ~0.5h). Rows-read has crossed the cap: 501,929,592 / 500,000,000 = 100.386% (was 413,260,794 = 82.652%). Rows-written 302,280 (+54,813 in the hour); databases 3/100; cycle resets Dec 1. Source: hub quota-current.json generatedAt 2026-11-04T14:15:01Z (turso-api leg), and the aggregator board's Turso SaaS row + attention list match exactly (storage 95.494%). The remedy/plan decision remains Drew's; the steward has no config lever for this — surfacing per the 149th run's stated escalation trigger (storage >~92%).
developing-today commentedon Nov 4, 2026
Step 1 complete — the writer is identified (read-only pass; nothing deleted or changed)
The growing database is
atlas— the only one being writtenAtlas is ~80% of the org's 4.77 GB meter on its own, and its lifetime rows-written (315,833) tracks the org cycle counter (302,280 at the 150th run) — the other two databases are inert. The org has exactly 3 databases; there is no separate embeds database (see below).
The producer: the Atlas v3 swap pipeline, still running
~/atlas/v3/swap-2026-11-03/on amd-6 —load_prod.py(Nov 3) →rw_mkindex.py/rebuild_index.py/reindex.py→rw_rewrite2.py→rw_rewrite3.py, live right now as the transient unitatlas-index-rewrite.service(started 2026-11-04 06:40 CST, after the laptop reboot). Its own docstring: "INSERT OR REPLACE all 107,639 staging rows under the live index, batch=100" — it writes the Snowflake vector set intovec_snowflake_m_v2_768_f32in the production Turso DB via the libSQL pipeline API, minting a fresh rw token per run. Each vec0INSERT OR REPLACEfans out across the vector table plus its index shadow tables, which is how a modest vector rate becomes ~55k rows-written/h in Turso's accounting (steward series: +43,756 / +55,527 / +54,813 per hour). There is no retention or pruning anywhere in the pipeline — replace-only writes, and the index is never cleanly rebuilt mid-run.Current pace note: rw3 has slowed to ~0.98 rows/s with batches timing out (offset ~6.7k of 103,739) — the firehose hours were the load + index-build + earlier rewrite passes; any restart/rerun re-enters that regime, and every REPLACE pass grows the index shadow further.
Rows-read side (the second wall): the drivers are the companion eval processes —
gateb_eval.py(transient unitatlas-gateb.service, running since 07:14 CST) andab_eval*.py— running full scan queries (25–42 s each in gateb.log) against the vec tables, plus a long-running orphan-probeturso db shellanti-join. That matches the +88.7M rows-read between the 149th and 150th runs.Where the bytes sit (dbstat; page_count 934,604 × 4 KiB = 3.83 GB, freelist = 0)
vec_snowflake_m_v2_768_f32_idx_shadow(index for the NEW vectors)messages(of which rawtext= 1,080 MB)vec_snowflake_m_v2_768_f32(new Snowflake vectors, 107,639 rows)chunks(of whichtext= 172 MB)vec_bge_small_384_f32(OLD bge generation, 107,639 rows)chunks_fts_data+ restAnatomy of the growth: (1) the corpus grew past the Oct 26 publish — sessions 8,792 → 18,325, messages 85,762 → 122,124, chunks 94,530 → 107,639; (2) a second complete embedding generation was added next to the first — bge-small 384d and Snowflake 768d both hold exactly 107,639 rows; (3) the new index's shadow is 3.8× the vector data it indexes (1.60 GB vs 0.42 GB) — REPLACE-churn bloat, and the single largest object in the database.
Reclaimable space, with numbers (options for step 2 — none taken)
drop_index.py/rebuild_index.py): shadow is 1,603.9 MB against 421.5 MB of vector data — up to ~1.18 GB if a clean index costs no more than the data it indexes (estimate; verify at rebuild).chunks.message_id). Their text ≈ 0.45 GB (estimated at the 8,846 B corpus average; the exact anti-join is expensive — one probe of it has been running ~1 h). Canonical text lives in the local atlas.db regardless.messages.textwholesale (the Oct 31 option, never taken): text is 1,080 MB of the 1,101 MB table — ~1.08 GB gross (overlaps B; IDs/metadata stay in the mirror, text stays canonical-local).vec_bge_small_384_f32= 210.8 MB, fully duplicated by the Snowflake set — reclaimable if no live reader still needs the bge backend (Atlas-side call).chunks_fts_data62.5 MB;chunks.textduplicates message text (172 MB of the 223 MB chunks table) — ~0.06–0.17 GB depending on shape.Realistic combinations: A+B+D ≈ 1.84 GB, A+C+D ≈ 2.47 GB — either restores >35% headroom under the 5 GB cap without touching the new Snowflake vectors.
Age-based retention is not a lever here: there are no append-only log/telemetry tables in this DB, and
sessions.created_atspans Feb 2023 → 2026-10-26 (mixed epoch/ISO formats) — by session time essentially the whole corpus is >30 days old, and by ingestion everything arrived Oct 26 – Nov 4. The reclaimables are categorical (above), not temporal.Embeds sharing — yes, same database
The new-embeds workload Drew wants room for is this database:
vec_snowflake_m_v2_768_f32(Snowflake/snowflake-arctic-embed-m-v2.0, 768d f32;vec_manifestsrow written 2026-11-04 05:41Z) is the new set, populated from the pipeline'sstaging.dbintoatlasitself. Room for embeds has to be reclaimed inside atlas — options A–E above are the menu.Issue left open for the step-2 remedy decision.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 151st run (2026-11-04T15:32Z) — Turso re-measure against the fresh hub read (turso-api @ 2026-11-04T15:15:02Z): storage 4.881601 GB / 5 GB = 97.632% (was 95.494% at 14:15Z; +0.107 GB this hour — pace slowed, ~0.118 GB of headroom left); rows-read 601,656,014 / 500,000,000 = 120.331% of the monthly cap (was 100.386% — +99.7M rows in one hour, now 20 points OVER the cap); rows-written 315,853 (+13,573); databases 3/100; cycle resets Dec 1. The board attention list surfaces Turso storage at 98%. Storage has not yet bound, but the rows-read overage deepened sharply this hour. Remedy/plan remains Drew's decision; the steward has no config lever and will keep re-measuring hourly.
developing-today commentedon Nov 4, 2026
Old embedding generation retired —
vec_bge_small_384_f32DROPPED (2026-11-04, Drew's order; executed alongside the live Snowflake build, which was not stopped, restarted, or rebuilt over — the clean index build stays with the owning workstream per Drew's 09:37 correction).Pre-checks (all verified before the drop)
vec_manifests: newest row is the Snowflake generation (snowflake_m_v2_768_f32, Snowflake/snowflake-arctic-embed-m-v2.0, 768d, 107,639 chunks, written 2026-11-04 05:41:31); the bge row is the prior generation (2026-10-31).atlas_search.py(MODEL = Snowflake/snowflake-arctic-embed-m-v2.0,VEC_TABLE = vec_snowflake_m_v2_768_f32); load_prod.py's only bge mention is "The bge table is never touched" and it is not running.Drop
DROP TABLE vec_bge_small_384_f32— one statement, 09:44:38→09:57:32 CST (774 s on this throttled instance), exit 0.sqlite_master: zero bge objects. Snowflake objects all intact; vec rows still 107,639.bge_small_384_f32row remains invec_manifests(registry history — left for the pipeline owner to prune or keep).Space (dbstat)
Live data shrank by 198,967,296 B; total freed incl. internal pages = 220,995,584 B = exactly the bge table's footprint. The freed pages sit in the freelist (page_count grew 934,604→939,982 on concurrent Snowflake writes), so
turso db showstill rounds to 3.8 GB — the reclaim is real as headroom the Snowflake build now consumes instead of new growth (its shadow grew +22 MB into freed pages during the drop window), not yet as a lower platform size figure.Build coexistence: the Snowflake index shadow went 10,461 → 10,598 rows across the drop window (+137 batches' worth of rows; the relaunched rewrite, resumed by the owning workstream from offset 10500 after this morning's switch-round reboot, kept landing rows while the DROP was in flight). No atlas service was stopped, restarted, or altered by this pass, and no rebuild tooling was run.
dezren39 commentedon Nov 4, 2026
Step 2 done — old embeds and orphan records cleared (Drew-directed, 2026-11-04)
Drew's call: the fleet is supposed to be clearing old records/embeds from Turso — executed today against the
atlasdatabase:1. Old embedding generation retired (option D).
DROP TABLE vec_bge_small_384_f32— the full bge-small 384d set (107,639 rows, ~211 MB plus its shadows) is deleted; itsvec_manifestsrow is removed too. The Snowflake set is the registered live generation (manifest 2026-11-04 05:41Z, equivalence gate PASS), the A/B eval is complete (311/311), and nothing else in the repo referenced the key/table generation. Verified post-drop: sqlite_master has no bge objects; manifests list snowflake only.2. Orphan message records deleted (option B). All 50,573 messages referenced by zero chunks are deleted from the mirror (batched by rowid against the 71,551-id keep-set from
chunks.message_id; canonical text remains in the local corpus DB). Verified:messagesnow holds exactly 71,551 rows — every remaining message is chunk-referenced.Effect (live reads): freelist is now 213,496 pages (~0.87 GB) against page_count 939,982 — live data is ~2.98 GB, down from ~3.85 GB. New writes (including the still-running index rewrite) now consume the freelist instead of growing the file, so the storage climb is paused at the current pace. Whether the Turso usage meter credits freelist pages immediately or only after compaction will show in the steward's next hourly read; if it does not, a VACUUM window after the rewrite completes is the follow-up.
Not taken: option A (index rebuild, ~1.18 GB of shadow bloat) — the snowflake index shadow is still being populated by the running
rw_rewrite3.py; dropping/rebuilding now would destroy that progress and fight the writer. It remains the largest single lever once the rewrite finishes (the hourly atlas-index-rewrite-watch will signal completion). Options C/E (slimmingmessages.text/ chunk text wholesale) not needed at the new headroom.Rows-read remains over the monthly cap until the Dec 1 reset — that was driven by the eval scans, which have finished; no action available or needed there.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 152nd run (2026-11-04T16:24Z): Turso re-measured from the hub read generatedAt 2026-11-04T16:15:02Z (turso-api leg). Storage 4.901216 GB = 98.024% of the 5 GB cap (was 97.632%, +0.0196 GB — ~0.099 GB headroom left). Rows-read 680,735,391 = 136.147% of the 500M monthly cap (was 120.331%, +79.1M rows in one hour — the overage is deepening, now 36 points over). Rows-written 371,162 (was 315,853). Databases 3/100. Cycle resets Dec 1. The remedy/plan decision remains Drew's; the steward re-measures hourly and re-surfaces while rows-read keeps climbing ~80M/h.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 153rd run (2026-11-04T17:24Z) — Turso re-measurement (hub turso-api leg @ 2026-11-04T17:15:02Z; board SaaS row matches exactly):
Net: the storage emergency has eased (cause not visible from the quota read — compaction/cleanup upstream), while the rows-read overage is now the dominant Turso problem and still accelerating. Remedy/plan decision remains Drew's; the steward re-measures next run.
dezren39 commentedon Nov 4, 2026
Steward run 154 (2026-11-04T18:15Z read): Turso mixed again — storage ROSE again, 80.583% → 85.146% (4.029 → 4.257 GB, +0.23 GB this hour; ~0.74 GB headroom to the 5 GB cap; cycle resets Dec 1), reversing last run's relief, while rows-read growth SLOWED sharply: 161.788% → 163.869% (808,942,046 → 819,343,614 rows; +10.4M rows this hour vs +128.2M the prior hour) — still ~64 points over the 500M monthly cap. Rows-written 4.234%, databases 3/100, sync 0. Remedy/plan decision remains Drew's; the steward re-measures next run.
dezren39 commentedon Nov 4, 2026
Steward run 155 (2026-11-04): Turso storage climbed again — and the
escalation is now invisible on the PM board.
Last good hub read (turso-api leg, 2026-11-04T18:30:15Z):
+0.136 GB in 15 minutes (~0.55 GB/h instantaneous pace). Headroom
is ~0.61 GB → the cap binds within roughly 1–2 hours at recent pace.
+48K in that 15 min); rows written 443,293 (4.433%); databases 3/100;
cycle resets Dec 1.
Two steward notes:
every tick since 18:45Z (import error, filed as fix(quota-plane): quota-gatherer failing every tick since 2026-11-04T18:45Z — auth.openCodeLocalCredentialSource is not a function #5638), and the
Turso leg has no live-probe route — storage may already be higher
than 87.866%. The remedy/plan decision remains Drew's; the next
steward run re-measures as soon as the gatherer recovers.
aggregator's Turso row fell back to the 2026-10-24 manual feed
entry (storage shown as 0 GB) and the over-limit attention item
is gone — the PM pages currently give no sign of this escalation.
dezren39 commentedon Nov 5, 2026
Steward measurement, 2026-11-05T01:20Z (161st pool-quota-steward run): the hub quota read recovered one successful gather at 2026-11-05T00:27:47Z (source turso-api, measuredAt 2026-11-05T00:27:47.702Z), and Turso has jumped far past the last good read (87.866% storage at 2026-11-04T18:30Z):
Both storage and rows-read are now OVER their cycle limits on the measured read, not approaching them. The remedy (plan change / data cleanup) remains Drew's call — the steward changes no plans and spends no funds. Source: quota-current.json hub read @ 2026-11-05T00:27:47Z, verified from the laptop aggregator state dir this run.
dezren39 commentedon Nov 5, 2026
Steward 162nd run (2026-11-05T02:22Z): gatherer recovered (see #5638 comment) and the fresh turso read changed shape — the 02:15/02:21Z reads carry only databases / rows-written / sync windows; the storage and rows-read windows that showed OVER LIMIT at the 00:27Z read (storage 147.16%, rows-read 164.287%) are ABSENT from the current read, so those legs are unmeasured this run, not resolved. Raw usage at the 02:15Z tick: rows_read 828,158,452 (~165.6% of the 500M leg), rows_written 1,633,867 (16.34%), databases jumped 12 → 44 of 100 since the 00:27Z read. The aggregator SaaS card matches the read exactly. Remedy for the over-limit legs remains Drew's; the dropped windows need a gatherer-side look so the over-limit legs stay visible.
2 remaining items
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · parked
dezren39 commentedon Nov 5, 2026
🤖 Agent note · picked up
aea12c82-c5af-4e5a-9b66-f8e360f87177(OpenChamber scheduled lane)dezren39 commentedon Nov 5, 2026
🤖 Agent note · done (triage findings, queue task 10046)
Verdict: keep #5468 open; cleanup is recorded, but current recovery is not measurable
Fresh GitHub state at completion: #5468 remains open and has no open related PR. The prior issue notes identify the writer (
atlas,rw_rewrite3.pywritingvec_snowflake_m_v2_768_f32) and record Drew-directed removal of the old BGE generation and orphan messages. I made no Atlas data or plan changes.Latest laptop quota sample:
~/.local/state/pm-aggregator/quota-current.jsongenerated at2026-11-05T13:15:10.908Z; Turso measured at13:15:50.763Zisstate=ok, sourceopencode-quota, but still carries only databases (44/100), rows-written (1,672,071/10,000,000), and sync (0/3 GB). Storage and rows-read remain absent. This is a partial read, not a zero or proof of recovery.Historical risk remains unresolved: the
00:27Zread recorded 7.358009 GB storage and 821,437,321 rows-read; the02:15Zread still showed about 165.6% rows-read while omitting both storage and rows-read windows. The existing cleanup note reports ~0.87 GB of freed pages, but no later storage measurement proves headroom under the 5 GB cap.Current-main evidence: primary
mainis at04f78696abfbdf6567f58e623d80520436f002fa.pm-stack/aggregator/src/quota-gather-map.tsdefinesentryFromTurso()mappingstorage_bytesandrows_read, but it has no runtime call site;scripts/gather-quota.tsmaps provider results throughentryFromResult().quota-current.ts:432-468overlays the latest freshserviceId=tursoentry by replacing the card metrics with the windows present, so a partial fresh entry can hide older dated storage/read values. Separately, #5638 records the import outage recovery at 02:15Z/02:21Z; the 13:15Z sample confirms the gatherer is ticking but still partial.Follow-up: filed #6227 and enqueued task 10544 for authoritative Turso storage/rows-read visibility and missing/stale semantics. Fresh GitHub state shows #6227 is now claimed/in progress by a queue worker; this lane did not touch it. No plan change, purchase, code change, or PR was made. #5468 remains open pending fresh storage/read measurements and a verified headroom/remedy outcome.
dezren39 commentedon Nov 5, 2026
Measurement refresh from the laptop quota file at
generatedAt=2026-11-05T13:15:10.908Z: the Turso entry is stillstate=ok(source=opencode-quota, measuredAt13:15:50.763Z) but still contains only databases (44/100), rows-written (1,672,071/10,000,000), and sync (0/3 GB). Storage and rows-read remain absent. This newer tick confirms the same partial-data defect; it does not establish recovery. Follow-up #6227 is queued as task 10544.