Skip to content

design: move ci.events webhook ledger from Supabase to Turso staging #5693

Open

Description

@dezren39

Design scope only — no data moves in this issue's first pass.

What moves

  • From: Supabase (muse-dev2) ci.events (10,952 live rows, 31.8 MB — ~2.9 KB/row of jsonb webhook payload)
  • To: Turso, staging tier, partitioned by day/repo

Why

A pure append-only CI webhook ledger: 10,957 inserts / 0 updates in the stats window since 2026-10-29, no foreign keys, read by scan (its one view, ci.recent_failures, is a filtered scan). It is the fastest-growing table in the project by bytes and has no transactional consumer — under the §9 placement rule it is a Turso stream, not Postgres state.

Design doc: §9 of doc/2026-11-04T21-27-34Z_design_atlas_scaler_revision/ (PR #5667, addendum PR #5689).

Risks

  • The webhook receiver is a live write path — cutover must not drop events during deploys (receiver-side buffering or dual-write).
  • recent_failures consumers (alerting? dashboards?) must be inventoried first; failure-visibility must not regress during the move.
  • Retention is currently unbounded; the Turso design should state a retention window rather than replicating unbounded growth.

Sketch of steps

  1. Inventory the receiver(s) and every reader of ci.events / ci.recent_failures.
  2. Design the Turso table + a recent_failures-equivalent query; set retention (propose 90 days hot, cold export beyond).
  3. Dual-write at the receiver, backfill the retention window, diff failure queries.
  4. Flip readers, stop the Postgres write, keep the Postgres table read-only for one retention window before any drop decision (separate approval).

Links: design doc PR #5667 · addendum PR #5689 · tracking dezren39/hatch-personal#21

Activity

  1. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 5, 2026
  2. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done (triage)

    Triage findings (verified fresh 2026-11-05):

    1. Design basis is on main. PR docs: Scaler-era Atlas/Turso data architecture revision #5667 (Scaler-era Atlas/Turso data architecture revision) and PR docs: Supabase placement audit addendum (§9) — what moves out of Postgres #5689 (Supabase placement audit addendum, §9) are both MERGED. The design doc doc/2026-11-04T21-27-34Z_design_atlas_scaler_revision/ names issue design: move ci.events webhook ledger from Supabase to Turso staging #5693 for ci.events: 10,952 rows / 31.8 MB, 10,957 inserts / 0 updates in the window — a pure append-only stream, matching the issue's placement argument.
    2. The inventory cannot be done by repo grep alone. Code search for ci.events and ci.recent_failures in developing-today/code hits ONLY the design doc — the webhook receiver and the view's consumers live outside this repo's code (Supabase-side objects and/or external alerting/dashboards). The follow-up's inventory step must enumerate Supabase-side views/functions and external consumers explicitly, with not-found results stated, not assumed.
    3. Gates preserved: this issue is design-scope only for its first pass (no data moves), and step 4's Postgres drop requires separate approval — the follow-up is scoped to steps 1–2 (inventory + Turso table / recent_failures-equivalent query / retention design, proposing the issue's 90-days-hot window) and explicitly excludes dual-write, backfill, and any cutover.

    Disposition: follow-up triage/design task enqueued in this pass (see addendum). Issue #5693 stays open for it.

  3. dezren39 commented on Nov 5, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · addendum to task 10129 triage

  4. added
    queue/queuedHas a queued task in the Supabase agent queue
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 5, 2026
  5. 40 remaining items

  6. dezren39 commented on Nov 8, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  7. developing-today commented on Nov 8, 2026

    @developing-today
    Contributor

    🤖 Agent note · handed off

  8. developing-today commented on Nov 8, 2026

    @developing-today
    Contributor

    🤖 Agent note · handed off

  9. added
    queue/claimedQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 8, 2026
  10. dezren39 commented on Nov 8, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  11. added
    queue/blockedQueue or user escalation state
    status/blockedQueue or user escalation state
    and removed
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 8, 2026
  12. dezren39 commented on Nov 8, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · blocked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

queue/blockedQueue or user escalation statestatus/blockedQueue or user escalation statestatus/triagedTriaged and routed (see agent note)task/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions