Repository navigation
Quota recorder: record on sandbox, ship via fleet proxy, ingest at a laptop broker (retire the recycle-vanishing sandbox units) #5703
Description
Quota recorder: move the Supabase write to the laptop (design + recommendation)
Current state (verified 2026-11-04)
- The sandbox quota recorder runs from systemd units installed by
~/tools/quota-recorder/(its timer + service). Like every/etc/systemd/systemunit, they vanish on every sandbox recycle and must be reinstalled — today that reinstall is a manual/cron-assisted step (the 2026-11-04 15:08 boot check found them missing again post-recycle and reinstalled them from~/tools/quota-recorder/, after which ticks resumed with Supabase acks). - The recorder writes usage snapshots to Supabase (muse-dev2) from the sandbox, which means the sandbox side holds/uses the Supabase write path directly.
- Recycles are frequent (six boots on 2026-11-04 alone), so "reinstall after recycle" is a recurring failure surface, and a missed reinstall = a silent gap in the quota record the burn plan depends on.
Proposal (Drew's question, 2026-11-04)
Split the recorder the same way the queue and secrets already work:
- Sandbox side: record-only. A small local writer appends quota observations to a local spool (JSONL under
~/tools/quota-recorder/state or~/.local/state/quota-recorder/). No Supabase credentials, no network write on the sandbox. Survives recycles by construction (the spool lives in/home/hatch, which persists). - Ship via the fleet proxy. Batches move sandbox → laptop through the existing fleet session proxy, the same transport every other broker call uses.
- Laptop side: broker-owned ingest. A laptop endpoint (a
fleet serverole alongside queue-broker :4801 / secrets-broker :4800, or an ingest route on an existing broker) owns the Supabase connection and performs the insert — credentials live only on the laptop, exactly like the queue-broker holding the Supabase credentials for queue operations today. - Ack + trim: the laptop acks batch ids; the sandbox trims its spool only on ack. A laptop-off window (e.g. 2026-11-04 evening, laptop powered down) becomes a growing local spool, not lost data — recording continues and ships when the laptop returns.
Evaluation vs. status quo
| Status quo (sandbox units → Supabase direct) | Proposed (spool → proxy → laptop broker) | |
|---|---|---|
| Survives recycle | No — units vanish; reinstall is a step that can be missed | Yes — no unit needed for recording; shipper can be an init-ensure-style prelude check or a tiny timer reinstalled by the same mechanism as other fleet units |
| Credentials on sandbox | Yes (Supabase write path) | No — laptop-only, matching the queue/secrets pattern |
| Laptop off | Recorder still writes (Supabase is remote) | Recording continues to spool; ingest pauses; no loss |
| Failure visibility | Missing ticks noticed by the boot check | Spool depth + last-ack age are directly measurable; alert on spool age |
| New machinery | — | One laptop ingest endpoint + batch/ack protocol (small; mirrors trip-log ingest follow-on on #5343) |
Recommendation
Adopt the proposal. The deciding factors: the recycle-vanish reinstall already failed once today and will keep failing on the same cadence; the credential placement matches the fleet's established broker pattern; and the spool design converts the laptop-off case from data loss risk into mere delay. The one real cost — ingest pauses while the laptop is off — is acceptable because the quota data is consumed on planning timescales (the burn plan reads snapshots, and the gatherer already tolerates staleness by design: a stale read never idles a pool).
Acceptance / next steps
- Local spool writer on the sandbox replacing the direct-write recorder; format = the existing snapshot payload, one JSON object per line, batch id per ship.
- Laptop ingest endpoint with ack; idempotent on batch id (replays after a crash are expected).
- Shipper on the sandbox (fleet-proxy transport) with spool trim-on-ack and a last-ack watermark.
- Boot check updated: instead of "reinstall the units," it verifies spool freshness + last ack; a stale ack surfaces like today's missing-timer finding.
- Old sandbox units retired once the new path has produced a week of clean acks.
Related: #2072 (unified quota attribution/observability) covers the read/attribution side; this issue is only about the recorder write path and where the Supabase connection lives.
Acceptance criteria
- The quota recorder runs on the sandbox and ships records via the fleet proxy to ingestion at a laptop broker, per the design in this issue.
- The retired Supabase path is removed or explicitly disabled, with no silent dual-writing.
- Recorded quota data matches the previous recorder's output for the same inputs (parity spot-check recorded on the issue).
Metadata
Metadata
Assignees
Labels
Large: multi-day or open-ended; expect staged check-insIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Queue task is parked (waiting on CI, time, or an event)Deliberately paused with a recorded resume conditionQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
dezren39 commentedon Nov 5, 2026
Intake review
Related: part of the quota-pipeline batch filed 2026-11-04 23:12–23:18Z (see also #5704). Notes: this retires a Supabase dependency for quota records in favor of sandbox-record + fleet-proxy ship + laptop-broker ingest. Risk: dual-running or a silent gap during cutover would corrupt the quota history the dispatch gates read. Suggested approach: build the ship/ingest path behind the current recorder, parity-check outputs, then retire the old path in the same change.
Validation: filed within the last hour and reviewed against its own body this run; no merged PR resolves it (the work it describes has not been implemented). Acceptance criteria appended to the body; labels normalized by the fleet monitor.
🤖 Agent note · intake enrichment
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
opencode— the laptop claimant launches the bound opencode session for task 10137, which drives it to a terminal state under this queue's worker id.145 remaining items
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 6, 2026
🤖 Agent note · parked