Restore procedure: reference tables + post-restore row-count diff gate

Origin: the 2026-11-07 fleet-queue restore-completeness audit (#6853, comment 6715020584), prompted by Drew's correction that the lane registry was supposed to be filled from the 2026-11-06 snapshot.

What happened: the fleet-queue cutover loaded its backup through a hand-built per-table export whose manifest listed five tables (tasks, leases, workers, checkpoints, repo_task_lane_ledger). agent.lanes — a reference table in the same queue core — was never exported, so the recreated store ran a full day with an empty lane registry and every lane claim rejected (unknown lane_id, #6853). The audit also found agent.token_expirations (10 rows) has no Turso home anywhere. Whole-database moves (fleet-platform, fleet-mirror) could not fail this way — they import every table by construction and had flip-count verification. Only selective per-table restores are exposed, and nothing in the procedure caught the omission: the load "matched the export exactly" because the export itself was incomplete.

Required changes to the recreate/restore procedure (any selective export → fresh store)

Source-inventory-first checklist. Before any export is built, enumerate EVERY table in the source scope (for a schema: information_schema / dump COPY headers) and record it in the export manifest with its row count. Every source table must appear in the manifest with an explicit disposition: load, archive (name the destination), or retire (name the reason). A source table missing from the manifest fails the export step — silence is not a disposition.
Reference/config tables are named explicitly. Registries and trackers (lanes-class: lane registries, worker registries, token/credential trackers, watermarks, feature/config tables) are the highest-risk omission class because data tables load fine without them and the failure surfaces later, elsewhere (here: a day of rejected claims). The checklist names this class and requires a second-person/second-lane read of the manifest against the source inventory.
Post-restore row-count diff gate. After load, an automated per-table diff (snapshot count vs live count, with an allowlist for explained deltas: post-cutover writes, by-design archives like agent_events_pg_history, pre-rename duplicate keys) must pass before the cutover is declared complete. Any table that is non-empty in the source and empty or unexplained-short in the target blocks the declaration. The 2026-11-07 audit's diff table on queue: claim_repo_task rejects lane openchamber-a (unknown lane_id) — OpenChamber hook worker cannot claim #6853 is the worked example.
Backfill path is additive-only. Gaps found by the gate are backfilled with INSERT OR IGNORE from the snapshot, before/after counts recorded (lanes precedent: 32 → 47, no deletes, no overwrites).

Named disposition owed: `agent.token_expirations`

10 rows (credential names, services, expiry/renew-by dates, renewal-watch cron names; never secret values) survive only in the 2026-11-06 snapshot (~/workspace/scratch/muse-dev2-queue-core-export-2026-11-06/latest-backup.pgdump.sql.zst on the laptop, and the throwaway PG restore from Lane C). Pick its Turso home and load it:

Recommendation: fleet-platform as agent_token_expirations, beside the other human-facing agent_* operational tables that landed there (agent_external_comms, agent_quota_records) — it is a human-maintained tracker read by renewal-watch jobs, not broker state.
Alternative: fleet-queue with the rest of the audit's queue-core class.
Either way: load the 10 snapshot rows, then repoint/note the renewal-watch consumers (the GHCR PAT renewal fires 2027-01-15 and the project-token renewal 2027-10-01 per the tracker's own rows) so the tracker is live again, not just preserved.

Acceptance

Restore/recreate runbook (doc/fleet + hatch-personal backup/restore docs) updated with (1)–(4).
The diff gate exists as a runnable script/query set, not just prose.
agent.token_expirations loaded into its chosen store; counts verified; consumers noted.

CODE-2223

token_expirations disposition — DONE (2026-11-07 18:35 CST, Drew's order: finish it)

The audit's one stopped-at gap is now closed:

Table created: Turso fleet-platform.agent_token_expirations, beside its siblings agent_external_comms (69 rows) and agent_quota_records. Schema = the PG original (migration 0023, agent.token_expirations) with type adaptation only: bigint id → INTEGER PRIMARY KEY, date/timestamptz → TEXT (the fleet-platform convention — every sibling table stores dates as TEXT), status CHECK constraint preserved (active/retired/check).
Rows loaded: 10/10 from the 2026-11-06 snapshot (fleet DIY backup of muse-dev2, 2026-11-06T10:47:11Z, latest-backup.pgdump.sql.zst), ids 1–10, extracted from the dump's COPY block (the queue-core per-table export never included this table — same export-manifest gap that dropped agent.lanes).
Readback proof (independent turso db shell queries): count(*) = 10, min(id)=1, max(id)=10; full listing matches the snapshot row-for-row (names, services, statuses). Spot-checks on the two live renewal watches:
- id 1 fleet-build GHCR publish (developing-today): expires_on 2027-02-01, renew_by 2027-01-15, watch cron ghcr-org-pat-renewal, fires Fri 2027-01-15 09:12 CST ✓
- id 2 fleet-project-access: expires_on 2027-10-26, renew_by 2027-10-01, watch cron github-project-token-renewal, fires Fri 2027-10-01 09:12 CDT ✓
- Long-text fidelity: id 3 notes length 451 chars in both snapshot and Turso, expires_at 2026-11-04 22:01:00+00 exact — no COPY-escape mangling.
Correction to the audit comment's parenthetical figures: the watch fire dates it cited (2027-01-15, 2027-10-01) are the rows' renew_by values and are correct; the expiry dates in that shorthand were not the snapshot's — the snapshot's actual expiries are 2027-02-01 (GHCR PAT) and 2027-10-26 (project token), which are what is now loaded and verified above. (2027-11-06 / 2027-11-04 in tonight's earlier summaries belong to other records/rows — e.g. row 3's expires_at is 2026-11-04 — not to these two credentials.)

No values were ever involved — names, dates, and storage locations only, per the table's own design.

🤖 Agent note · picked up

Agent: queue worker (repo-worker-code), lane code-t
Session / ID: task 11438
Found via: queue claim
Task: task 11438 (triage, issue Restore procedure: reference tables + post-restore row-count diff gate #6994) claimed; execution-lane assignment returned assigned_lane=direct, so this lane triages it directly. Revalidating fresh: issue Restore procedure: reference tables + post-restore row-count diff gate #6994 OPEN, no labels, no hard-stop labels.

🤖 Agent note · triage complete

Agent: queue worker (repo-worker-code), lane code-t
Session / ID: task 11438
Verdict: CONFIRMED, PARTIALLY DONE — one of three acceptance items is complete; the other two have no artifact at current main.

Fresh evidence (2026-11-08T01:19–01:21Z):

Acceptance item 3 (agent.token_expirations disposition) is DONE: the 2026-11-08T00:36Z comment on this issue documents agent_token_expirations created in Turso fleet-platform with 10/10 snapshot rows loaded and independent readback (count=10, ids 1–10, row-for-row spot checks, both renewal watches verified with corrected expiry dates 2027-02-01 / 2027-10-26).
Acceptance items 1–2 are NOT landed: the doc/fleet/ listing at current main has no restore/recreate runbook (nearest doc, queue-turso-cutover.md, greps clean for the inventory checklist / disposition manifest / row-count diff gate — only an unrelated outbox-manifest mention), and code search for the diff-gate and restore-procedure terms returns no script artifact. The procedure that dropped agent.lanes + agent.token_expirations in the queue: claim_repo_task rejects lane openchamber-a (unknown lane_id) — OpenChamber hook worker cannot claim #6853 cutover is still prose-free and unscripted.

Routing: follow-up task 11467 enqueued in repo-health-code-pilot — small_fix scoped to (1) the doc/fleet restore runbook covering all four required changes (inventory-first checklist with per-table load/archive/retire dispositions, reference/config class named with a second-reader manifest check, row-count diff gate with explained-delta allowlist, additive-only backfill) and (2) the diff gate as a runnable script/query set referenced from the runbook. This triage task (task 11438) completes as triaged; issue #6994 stays open pending task 11467.

🤖 Agent note · done

Agent: queue worker (repo-worker-code), lane code-t
Session / ID: task 11438
Task: task 11438 completed as triaged. Follow-up task 11467 (runbook + runnable diff gate) is queued; labels now reflect queue/queued + status/triaged for that follow-up. Issue Restore procedure: reference tables + post-restore row-count diff gate #6994 stays open.

🤖 Agent note · handed off

Agent: repo queue worker, lane code-y
Task: task 11467 handed off to the opencode claimant per execution-lane assignment (checkpoint stage handed_off). The claimant's bound opencode session drives this task to its terminal state.

🤖 Agent note · handed off

Agent: fleet monitor (repo-fleet-monitor)
Session / ID: task 11467 · repo-fleet-monitor run 2026-11-08 10:12 CST
Found via: monitor reconciliation
Task: Claim evaporated — task 11467 returned to queued and unclaimed in repo-health-code-pilot after the holding run ended without a terminal verb. Labels reset to the truth: queue/queued + status/needs-triage.
Expectation: next claimant picks the task up from the queue and re-applies claim labels on pickup.

Navigation Menu

Restore procedure: reference tables + post-restore row-count diff gate #6994