Repository navigation
fleet-queue: port the pg_cron sweepers (expire standalone, parked revalidation, events prune) #6803
Description
Port the PG pg_cron queue sweepers to the Turso backend
Follow-up from the Turso queue cutover (#6741, runbook doc/fleet/queue-turso-cutover.md deviations 3–5). On Postgres, pg_cron ran three queue jobs inside muse-dev2; the Turso backend covers only the first, and only lazily:
expire_leases— PORTED, but it runs insideclaim_repo_task(as in the PG function body). PG also ran it standalone every minute (agent-expire-leases). Effect of the gap: expired leases sit until the next claim against any queue (expire is global, so any claim sweeps all queues — the practical gap is small but real when no claims happen).revalidate_parked_tasks(daily,park-revalidation-daily) — NOT ported: parked tasks whose ready conditions are non-time-based never auto-release in Turso mode. Time parks already release through claim atnot_before.prune_events(daily,agent-prune-events, 30-day retention) — NOT ported: the Tursoeventstable grows unbounded.
Work: add a broker-side periodic task (or a small systemd timer driving a fleet verb) that runs expire_leases on a cadence, revalidates parked tasks per the PG ready-condition semantics (recovered shapes are in the park implementation in turso_queue.rs), and prunes events older than 30 days. Keep the semantics identical to the PG bodies where they were recovered; note any deviation on this issue.
Acceptance: with the sweeper live, (a) a task claimed with a short lease returns to queued without any intervening claim, (b) a parked task with a satisfiable non-time ready condition releases, (c) events rows older than 30 days are deleted and newer rows untouched — each proven against the live fleet-queue DB with readback.
Acceptance criteria
- A broker-side periodic mechanism runs lease expiry on a schedule (not only lazily inside claim), demonstrated by an expired lease being swept while no claims occur
- Parked tasks with non-time ready conditions are revalidated on a schedule in Turso mode, or the gap is explicitly accepted and recorded with reasoning
- The Turso events table is pruned to the 30-day retention the PG agent-prune-events job enforced, with the prune observable in logs/metrics
Activity
developing-today commentedon Nov 7, 2026
CODE-2093
dezren39 commentedon Nov 7, 2026
Intake review
Related: issue #6741 (Turso queue cutover; runbook deviations 3–5 per the issue body), issue #6838 (sibling missing-verb gap from the same cutover). Notes: item 1's practical gap is small (any claim sweeps all queues) — items 2 and 3 are the load-bearing ones; the events table grows unbounded until item 3 lands. Suggested approach: one systemd timer driving a fleet verb, mirroring the PG pg_cron cadences (1m expire, daily revalidate, daily prune).
— Fleet monitor intake (repo-fleet-monitor, 2026-11-07 run): acceptance criteria appended to the issue body, labels normalized (kind/infrastructure, priority/normal, effort/m, status/needs-triage, meta/enriched), marked meta/enriched. No merged PR or commit was found this run that explicitly resolves this issue, so it stays open.
dezren39 commentedon Nov 7, 2026
F4 linkage from the #5690 flip checklist (Lane B, 2026-11-07): the broker-side revalidator sweeper in this issue is checklist item F4 (and the events-prune port is F7). Verified today that only the
/events-seenprimitive exists in the broker code — there is no sweeper job or observe mode yet, so there was nothing to enable in observe-only mode during the fleet-fetch sink flip.Two things that raise this item's weight since filing:
AGENT_EVENTS_SINKwas flipped totursotoday (2026-11-07T16:44:52Z, proof on design: move agent append streams (agent.events, agent.quota_records) from Supabase to Turso staging #5690) —snapshot_refreshrows now land only in the fleet-queue Turso stream, soevent_seenrevalidation against the PG stream can never see them.park-revalidation-dailycannot substitute: PG (muse-dev2) has been down since 2026-11-06 19:25Z (infra: repo-queue PostgREST timeout: global failures lane-visible again — recurrence after #6450 closed (2026-11-06 21:15-21:31Z) #6619), and on its return it would read a PGagent.eventsstream that is missing every turso-only row.Ask for the implementation: first land in observe-only mode (log would-releases) for one daily cycle, per procedure C3 in
doc/fleet/agent-events-turso.md, before enabling releases.Merge pull request #6986 from developing-today/lane/6803-turso-sweepers
dezren39 commentedon Nov 8, 2026
Shipped 2026-11-07 (overnight lane 3) — PR #6986 merged; sweepers live on the Turso fleet-queue.
What runs now
POST /sweep(expire / prune / revalidate) — broker binary rebuilt from the PR branch,queue-broker.servicerestarted.fleet-queue-expire.timer— every minute (first timer-fired cycle observed 18:09:00 CST,expired_leases: 0that minute).fleet-queue-sweeper.timer— daily 05:17 CST + delay (next: Sun 2026-11-08 ~05:20). Canonical definitions inhome/common/default.nix(this PR); the live units are hand-installed copies until the next home-manager switch.Proof counts (live fleet-queue, seeded fixtures, cleaned up)
expired_leases: 1for a 5s-lease fixture →queued, binding cleared, lease row deleted, no claim involved.park_releasedevents; time park confirmed, stayed parked.events_prunedself-logged. Real deletions begin once cutover-era rows age past 30 days.One gated step remains (by design, procedure C3): the revalidator runs observe-only. The observe data says enabling is safe (0 real would-releases). After the first scheduled cycle (Sun 2026-11-08 ~05:20 CST) is eyeballed, enable releases with:
echo enabled > ~/.config/queue-broker/revalidator-mode(laptop; takes effect on the next daily cycle, no restart).