design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging

Design scope only — no data moves in this issue's first pass.

What moves

From: Supabase (muse-dev2) agent.events (158,802 live rows, 65.5 MB) and agent.quota_records (8,163 live rows, 3.9 MB)
To: Turso, staging tier (per §4 of the design doc), partitioned by day/work unit

Why

Both are append-only ledgers: in the stats window since 2026-10-29, agent.events shows 159,546 inserts / 0 updates / 3 deletes and quota_records is insert-only periodic plan/quota snapshots with a jsonb payload. The Scaler-era revision (§7 Phase A, §9.2) names exactly this shape — high-write append ledgers — as Turso staging-tier natives. Neither needs cross-row transactions or FK integrity.

Design doc: §9 of doc/2026-11-04T21-27-34Z_design_atlas_scaler_revision/ (PR #5667, addendum PR #5689).

Risks

agent.events is the heaviest-read table in the project (54M cumulative index scans since 2026-10-29) — its consumers are NOT yet audited; a blind move breaks whatever reads it (queue observability is the suspected reader).
Writers are live fleet systems (queue-broker path); dual-write cutover must not lose or reorder events.
quota_records.payload is jsonb — Turso side needs a JSON-column convention consistent with the staging schema versioning rule (§4.1).

Sketch of steps

Consumer audit: enumerate every reader/writer of both tables (code search + pg_stat_statements).
Define the staging-tier tables under the versioned staging schema; decide retention (events are a ledger — propose a rolling window with cold export).
Dual-write from the producers, backfill, compare counts/checksums per day partition.
Flip readers, observe one full fleet cycle, then stop the Postgres writes; keep the Postgres tables read-only for one retention window before any drop decision (separate approval).

Links: design doc PR #5667 · addendum PR #5689 · tracking dezren39/hatch-personal#21

docs: link the §9 migration design issues (#5690, #5691, #5692, #5693, …

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-w
Session / ID: task 10126
Found via: queue claim
Task: triage for issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 — assigned execution lane opencode; handing off to the opencode claimant, not worked directly by this lane
Expectation: claimant launches the bound opencode session and drives task 10126 to a terminal state; issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 verified OPEN with no hard-stop labels at handoff
Links: issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690

Task 10126 (lane code-s) claimed and assigned to the opencode execution lane. Handed off to the opencode claimant — the bound session drives this task to its terminal state. Parked by this lane per the handoff protocol.

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-n
Session / ID: task 10126
Found via: queue claim (re-claim after prior handoff park expired unbound)
Task: Triage design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging (design) — execution lane assigned opencode; handing off to the laptop opencode claimant, which launches the bound session. Verified fresh: issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 is OPEN with no hard-stop labels (no held, user/owned, do-not-merge, do-not-auto-edit).
Expectation: Claimant's bound opencode session triages issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 at current main and records blockers/ordering or routes the fix; if unbound by park expiry, task 10126 returns to the queue.
Links: task 10126 (repo-health-code-pilot)

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-y
Session / ID: task 10126
Found via: queue claim
Task: triage for issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 — handed off to opencode claimant (assigned_lane=opencode, session_id null at handoff); claimant launches the bound opencode session
Expectation: claimant session works issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 at current main
Links: task 10126

🤖 Agent note · handed off

Agent: repo queue worker, lane code-x (repo-worker-code)
Session / ID: cron repo-queue-worker-shift-x run 2026-11-08 11:19 CST; task 10126
Found via: claimed task 10126 this shift; execution-lane assignment returned opencode
Task: Claimed task 10126 for issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 (triage, attempt 10/12). Fresh check: issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 is OPEN, no hard-stop labels (held / user/owned / do-not-merge / do-not-auto-edit). Handed off to the opencode claimant — the claimant launches the bound opencode session; this lane does not work it directly.
Expectation: claimant session drives task 10126 to a terminal state; if it does not bind, task 10126 returns to the queue on lease expiry.
Links: task 10126 · issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-j
Session / ID: task 10126
Found via: queue claim
Task: Handing task 10126 (triage of issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690) to the opencode claimant; this lane does not work it directly.
Expectation: claimant launches the bound opencode session and records the terminal outcome for this triage; a triage verdict was already posted on issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 on 2026-11-08 (comment 6668124847) and is carried forward. Fresh revalidation by lane code-j at handoff (via the developing.today App REST route; the user-token GraphQL quota is currently exceeded): issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 OPEN, no hard-stop labels (held, user/owned, do-not-merge, do-not-auto-edit). Note: task 10126 is at attempt 11 of 12 — near its cap; if the claimant does not bind it, it will need a cap raise or manual disposition.
Links: issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690

🤖 Agent note · handed off

Agent: repo queue worker, lane code-o (repo-worker-code)
Session / ID: cron repo-queue-worker-shift-o run 2026-11-08 13:1x CST; task 10126
Found via: claimed task 10126 this shift; execution-lane assignment returned opencode
Task: task 10126 for issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 assigned to the opencode execution lane; handing off to the laptop claimant, this lane does not work it directly.
Evidence: issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 OPEN, verified fresh via gh; labels carry no held / user/owned / do-not-merge / do-not-auto-edit hard stop.
Next: claimant launches the bound opencode session; task 10126 parked 1200s for the claimant window.

🤖 Agent note · triaged (terminal)

Agent: queue worker (repo-worker-code), OpenChamber scheduled lane c8b9355e-319a-4640-8096-7ad19480d151
Session / ID: task 10126 · OpenCode session ses_e5e43533e9fffeelOYYBeNDjQo
Found via: atomic repo-queue claim

Terminal outcome: triaged. The substantive triage verdict already posted on this issue at 2026-11-08T09:56Z stands — this note records the terminal state and re-verifies it fresh rather than re-triage from scratch (resume-don't-restart).

Fresh re-verification (2026-11-08 19:41Z):

The verdict's design + build record is on main: doc/fleet/agent-events-turso.md.
The Turso sink is live and growing: events in fleet-queue held 61,755 rows, oldest 2026-11-07T05:07:08Z, newest 2026-11-08T19:40:51Z — 30,359 rows at the 09:54Z verdict, so ~31k rows appended in ~10h. The sink is running in production.
What remains is unchanged and gated: C4 (historical export at a watermark) needs a green PG window or a verified backup restore, and the PG-side decommission decisions are Drew's call after C4 verifies (doc §"What remains gated on PG recovery").

Why no new packet and no user/escalated: the remaining gate is infrastructure availability (PG recovery) plus a decision already recorded in the design doc — neither is a fresh Drew escalation, and no queue task can advance either. Filing a triage or fix packet would duplicate work already built and running.

Labels: all status/*, queue/*, work/* labels removed; task/triage (issue type) retained.

🤖 Agent note · picked up

Agent: queue worker (repo-worker-code), lane openchamber-scheduled-lane (1a180ae5-384f-4e71-8d77-08f7b1652c75)
Session / ID: task 13950 (ses_e5e422dba69ffeFBwEJYsVMveo)
Found via: queue claim
Task: revalidate the coverage-delta task for issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 against the latest terminal triage note before routing or editing.
Expectation: use current repository/GitHub state; resume the prior triage outcome rather than duplicate it if acceptance is already met.
Links: task 13950 · issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690

🤖 Agent note · done (already satisfied)

Agent: queue worker (repo-worker-code), lane openchamber-scheduled-lane (1a180ae5-384f-4e71-8d77-08f7b1652c75)
Session / ID: task 13950 (ses_e5e422dba69ffeFBwEJYsVMveo)
Found via: queue claim
Task: terminalized the stale coverage-delta packet by resuming the existing triage outcome; no source changes or new packet.
Evidence: issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 remains OPEN. Its latest triage note (2026-11-08 19:41Z) says design/build work is DONE and remaining scope is gated on Postgres recovery and a Drew decision. Fresh PR search found merged fleet-queue: agent.events Turso sink verbs + health terminal counters (#5690) #6762 (agent.events Turso sink) and gather-quota: cut quota_records sink from Supabase to Turso (Wave 3, #5690) #6655 (quota_records Turso sink), with related merged PRs external-comms: repoint ledger writer from Supabase PG to Turso fleet-platform #6910, github-ci Worker: webhook receiver re-home (#5693) #6921, and github-ci: public broker route for the events sink (#6925) #6984.
Links: task 13950 · issue design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690 · PR fleet-queue: agent.events Turso sink verbs + health terminal counters (#5690) #6762 · PR gather-quota: cut quota_records sink from Supabase to Turso (Wave 3, #5690) #6655

Navigation Menu

Repository navigation

design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690

What moves

Why

Risks

Sketch of steps

186 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690

Description

What moves

Why

Risks

Sketch of steps

Activity

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

186 remaining items

dezren39 commented on Nov 8, 2026

developing-today commented on Nov 8, 2026

dezren39 commented on Nov 8, 2026

dezren39 commented on Nov 8, 2026

dezren39 commented on Nov 9, 2026

dezren39 commented on Nov 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions