Skip to content

Turso replacement for the queue core: dedicated muse-queue DB + fleet queue-broker Turso backend #6741

New issue
New issue
Closed
Closed
Turso replacement for the queue core: dedicated muse-queue DB + fleet queue-broker Turso backend#6741
Labels
status:done

Description

@dezren39
dezren39
opened on Nov 7, 2026

Drew 2026-11-06 21:34 CST: "where is the turso replacement" — under his 2026-11-06 19:08 directive (escalate the Turso migration immediately; stand up dedicated Turso).

The 2026-11-06 migration moved shadows/mirrors/quota/ci to Turso + D1, but the queue core (agent.tasks, leases, workers, checkpoints, lanes, repo_task_lane_ledger) was deliberately kept on Postgres (#5689 audit) — and muse-dev2's data plane is down (Supabase platform fault, #6619), so the queue core has no live home. The GitHub fallback mode (#6654) is carrying work meanwhile.

This issue: stand up a dedicated Turso DB (muse-queue), reconstruct the queue-core schema from the recovered PG function bodies (pm-stack/opencode-claimant/migration-0021..0024.sql) + the broker/client contract, implement the queue verbs against Turso in fleet-queue behind a backend config switch (Postgres stays the default until cutover), prove a live claim/heartbeat/checkpoint/complete cycle with concurrent-claim refusal, and write the cutover runbook. Records on #6619 and #5690.

Activity

  1. developing-today commented on Nov 7, 2026

    @developing-today
    developing-todayBot
    on Nov 7, 2026 – with developing.today
    Contributor

    CODE-2083

  2. developing-today
    added
    status:in-review
    on Nov 7, 2026
  3. dezren39
    added 3 commits that reference this issue on Nov 7, 2026

    runbook: decommission-only end state, backup load step, simplified po…

    0d66f51

    fleet-queue: PG-true schema corrections, no-double-serve guard, perma…

    48a5855

    Merge pull request #6755 from queue/6741-turso-backend

    b88bb02
  4. developing-today
    added
    status:done
    and removed
    status:in-review
    on Nov 7, 2026
  5. developing-today
    closed this as completedon Nov 7, 2026
  6. developing-today
    added
    status:in-review
    and removed
    status:done
    on Nov 7, 2026
  7. developing-today
    reopened this on Nov 7, 2026
  8. dezren39
    added a commit that references this issue on Nov 7, 2026

    Merge pull request #6801 from queue/6741-runbook-record

    df74ca4
  9. developing-today
    added
    status:done
    and removed
    status:in-review
    on Nov 7, 2026
  10. developing-today
    closed this as completedon Nov 7, 2026

  11. dezren39 commented on Nov 7, 2026

    @dezren39
    dezren39
    on Nov 7, 2026
    MemberAuthor

    Turso queue replacement — COMPLETE; cutover executed 2026-11-07T05:10Z

    Database: fleet-queue (org dezren39, group great-star-lord, aws-us-east-1, id 01a1b43a-fb10-763c-ac90-5c1728b83906, turso://fleet-queue-dezren39.aws-us-east-1.turso.io, Type: Turso). Created FRESH with turso db create fleet-queue --group great-star-lord --tursodb per Drew's standard (2026-11-06 22:13 CST; fleet standard: atlas scaler revision design §6 — new databases are born --tursodb). The first instance (muse-queue, created without the flag → Type: SQLite / libsql mode) was destroyed and recreated fresh under the fleet-queue name on Drew's 22:14–22:16 rulings — never branched or converted in place. RW token vaulted in BWS as turso.fleet-queue-rw-token, verified byte-identical by readback + live pipeline query; the retired turso.muse-queue-rw-token secret was deleted.

    Implementation: PR #6755 merged b88bb02f (turso_queue.rs backend behind the existing broker protocol; QUEUE_BROKER_BACKEND=turso selects it, Postgres remains the code default). Schema reconstructed from the recovered PG function bodies (migrations 0021–0024) + the broker/client contract; workers/checkpoints/ledger shapes corrected against the real backup export; full deviation list (12 items) is in the runbook, doc/fleet/queue-turso-cutover.md (record updated by PR #6801, merged df74ca48). fleet-queue tests: 82 + 2 green at build time.

    Backup load (verified): the 2026-11-06T10:47:11Z R2 backup export (record: #6619 comment 6701907139) loaded as-is — tasks 8,170 · leases 107 · workers 34 · checkpoints 2 · repo_task_lane_ledger 3,347 = 11,660 rows, counts matching the export exactly; PG ids preserved (max 10,996). The 108 backup-era claimed rows carried expired leases; the first claims' expire_leases returned them to queued (post-cutover health: claimed = 0 in every queue).

    Live proof on fleet-queue (2026-11-07T05:08Z, dedicated fleet-queue-proof queue, all states read back from the DB): task 10997 full cycle (claim A → B refused exit 5 → lane opencode → bind → B heartbeat refused → A heartbeat/checkpoint KV row → complete done); task 10998 lease-expiry reclaim (attempts 2, lease_expired event); task 11000 refusal pair. Guard: with code#6646 live-claimed in the GitHub fallback, the Turso claim for task 10999 auto-blocked — {"task": null, "guard": "fallback_held", "held_by": "guard-holder-6741"}, exit 5; row blocked with reason_class: fallback_held; test task cancelled with reason, fallback claim released. Failover: client against the dead-PG production broker → fallback-switch JSON + exit 6; client against a killed broker → same, immediately.

    Cutover: ~/.local/bin/fleet = the merged build (previous binary preserved as fleet.bak-20261106-fallback); drop-in queue-broker.service.d/30-turso-queue.conf sets QUEUE_BROKER_BACKEND=turso; unit restarted by name — journal: queue backend: turso (https://fleet-queue-dezren39.aws-us-east-1.turso.io). Production-path verification: health names the Turso host + restored queues; production proof task 11001 claimed/completed by cutover-proof.

    Backfill from the fallback (master plan): outbox manifest (33) re-verified per target — 23 enqueued (tasks 11002–11024), 10 skipped (targets closed: code#6492, #6588, #6593, #6594, #6608, #6630, #6631, #6679, #6680, #6734). Four packets (openchamber#1737, opencode#1883/#1887/#1888) carried authoring defects (off-scale priorities; #1737 also kind small-fix + small-fix: dedupe family) and were enqueued from minimally corrected copies (priority → 70; kind → small_fix; family → fix:); prepared originals untouched — the outbox lane owns fixing them. The PG gap-backfill is a manual footnote only if PG ever returns (PG wins on conflict); muse-dev2's end state is DECOMMISSION, never delete.

    Fallback stays armed permanently: any primary-store failure on a work verb emits the fallback switch (exit 6) — verified live both ways (proof above).

    Incident note (honest record): during the first proof window, a pkill -f 'fleet serve queue-broker' aimed at the side broker also matched the production broker's argv and stopped it; it was restored within minutes (systemctl --user start queue-broker.service, verified active, PG backend state unchanged, no data lost — the broker is stateless against its DB). Standing lesson applied for the rest of this lane: processes are killed by recorded PID or unit name only.

    Follow-ups: prune_events + revalidate_parked_tasks sweeper port (filed separately); durable nix pin for the fleet build waits on #6725 (see #6658); the agent.events sink (PR #6762) was verified against the DESTROYED muse-queue instance — its AGENT_EVENTS_TURSO_* secrets need re-pointing at fleet-queue + live re-verification.

  12. dezren39
    added a commit that references this issue on Nov 7, 2026

    docs(fleet): agent.events sink target is fleet-queue after the #6741 …

    9e6fa1c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status:done

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions