Skip to content

program: finish Tier-2 execution migration to the laptop (unlock: delete sandbox fleet clones) #1723

New issue
New issue
Open
Open
program: finish Tier-2 execution migration to the laptop (unlock: delete sandbox fleet clones)#1723
Assignees
dezren39
Labels
effort/lLarge: multi-day or open-ended; expect staged check-inskind/infrastructureInfrastructure / host-portability workmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobtask/triageQueue packet kind: verify, reproduce, and route this itemuser/escalatedQueue or user escalation state

Description

@dezren39
dezren39
opened on Oct 28, 2026

Program

Finish moving the sandbox's remaining execution jobs to the laptop, per
the 2026-10-28 control-vs-execution classification
(hatch-personal: workspace/job-classification-2026-10-28.md). Control jobs
(~52: creators, sweeps, pr-diff, readouts, pool stewardship) stay on the
sandbox by design. Tier-1 queue shifts already execute on the laptop (only
their thin dispatchers sit on the sandbox — leaving them is deliberate).

Items

  1. agent-queue-worker (cron, 2h, pilot) — fold into the laptop shift
    pattern. Trigger: the repo-queue pilot interim check (2026-10-28
    17:25 CDT) verdict.     Do not migrate mid-pilot.
  2. cold-store-nightly-backup (daily) + cold-store-restore-drill
    (monthly/quarterly) — Cloudflare D1/R2 data duties. Vehicle: laptop
    systemd timers defined in the NixOS flake (the "all laptop services in
    the flake" pass), following the pattern proven by supabase-backup
    (script in ~/tools, timer, one verified run, then remove the sandbox
    cron). Laptop prerequisites verified 2026-10-28: wrangler + node are
    system packages there.
  3. supabase-turso-coord-mirror (daily) — Supabase→Turso sync;
    cloud-to-cloud. Same flake pass. Verified 2026-10-28: its tooling
    (~/tools/turso-exec.py, supabase-exec.py, goal-workspace state) does
    NOT depend on the retired ~/workspace/supabase tree.
  4. OpenChamber session long tail — sessions that still land in sandbox
    worktrees. Drains as the queue pilot concludes; no separate action.

Unlock condition (the point of the program)

When 1–4 are done and the snapshot-first control jobs (creators,
pr-diff-review, issue-sweep-rotating) are re-pointed from ~/git/fleet
snapshots to gh-API discovery or laptop-side snapshots, the sandbox fleet
clones + worktree farm can be deleted (preconditions verified 2026-10-28:
zettel clone has 1 unpushed commit, nix clone 1 dirty file, ~148
worktrees attached — resolve those first).

Done when

  • agent-queue-worker dispositioned by the pilot verdict and, if the
    pilot continues, running laptop-side
  • cold-store backup + drill running from laptop timers, one verified
    run each, sandbox crons removed
  • coord-mirror running from the laptop, one verified sync, sandbox
    cron removed
  • snapshot-first jobs re-pointed; sandbox fleet clones deleted

Acceptance criteria

  • agent-queue-worker is dispositioned by the repo-queue pilot interim-check verdict (2026-10-28 17:25 CDT) and, if the pilot continues, runs laptop-side.
  • cold-store-nightly-backup and the restore drill run from laptop timers with one verified run each; the sandbox crons are removed.
  • supabase-turso-coord-mirror runs from the laptop with one verified sync; the sandbox cron is removed.
  • Snapshot-first control jobs are re-pointed off the sandbox fleet clones, the clones' outstanding state (1 unpushed zettel commit, 1 dirty nix file, ~148 worktrees) is resolved, and the sandbox fleet clones are deleted.

Activity

  1. developing-today commented on Oct 28, 2026

    @developing-today
    developing-todayBot
    on Oct 28, 2026 – with developing.today
    Contributor

    CODE-693

  2. dezren39
    added
    kind/infrastructureInfrastructure / host-portability work
    priority/normalOrdinary priority (the default)
    effort/lLarge: multi-day or open-ended; expect staged check-ins
    status/needs-triageNot yet triaged; needs a first look
    meta/epicEpic / bundle of related work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 28, 2026

  3. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    MemberAuthor

    Intake review

    • Related: Gated on the repo-queue pilot interim check (today, 2026-10-28 17:25 CDT) for item 1; the classification source is workspace/job-classification-2026-10-28.md in hatch-personal. Labelled meta/epic — this is a program tracker, not a single fix.
    • Notes: Intake only; the body already carries items, an unlock condition, and a Done-when list, which the monitor has formalized as an Acceptance criteria section. Effort/l reflects the multi-item program.
    • Risks: Item 1 must not move before the pilot verdict (mid-pilot migration would contaminate the Phase 2 measurement). The unlock step deletes sandbox clones — the listed preconditions (unpushed zettel commit, dirty nix file, ~148 worktrees) are destructive if skipped.
    • Suggested approach: Work the items in order after the interim-check verdict lands; keep per-item progress as comments here rather than per-item issues, per the fleet's one-issue-per-set rule.

  4. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    MemberAuthor

    Progress 2026-10-28 (same day, at Drew's direction to start immediately):

    • Item 3 (coord-mirror): DONE. Script + state ported to laptop ~/tools/supabase-mirror/, daily systemd timer (06:12 CDT), sandbox cron removed. Verified with a real sync: workers 32, tasks 736, events_new 2690, last_event_id 20842. Porting surfaced and fixed three latent bugs: the script lived in the retired ~/workspace/supabase tree (would have broken the 10-29 run), its helper turso-exec.py was sandbox-only with a hardcoded /home/hatch token path, and a missing import os in that helper (fixed in the repo copy too).
    • Item 2 (cold-store): DONE. Nightly: wrapper + timer (03:12 CDT) on the laptop, Drive credentials ported, verified full run (all steps ok; D1 filed, manifest synced, 14 objects cold, within budget). Restore drill: wrapper with the Jan/Apr/Jul/Oct gate + monthly timer; verification run PASSED against the backup the laptop nightly had just produced (rehydration sha256 verified; D1 backup a content-equal subset of live). Both sandbox crons removed.
    • Item 1 (agent-queue-worker): still gated on today's 17:25 pilot interim verdict, as planned.
    • Item 4 + unlock: unchanged, drain with the pilot.

    Remaining on this issue: item 1 disposition, item 4, and the snapshot re-pointing + clone deletion.

  5. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    MemberAuthor

    Update 2026-10-28 ~11:50 CDT:

    • Mirror RETIRED (item 3 closed by retirement, not migration): Drew's call — the twice-daily Supabase pg_dump backup to R2 covers the durability concern, and the mirror was observation-only groundwork for a possible queue move to Turso (no active move plan; rebuild takes a day if it revives). Laptop timer disabled + units removed; the Turso muse-meta copy is frozen at 32 workers / 736 tasks / 20,842 events. The sync script stays in ~/tools/supabase-mirror/ (and in hatch-personal laptop/tools/) for that rebuild.
    • All-services-in-nix (Drew, same day): the three surviving migrated services (supabase-backup, cold-store-nightly, cold-store-drill) were hand-written user units — now declared in the flake: laptop-backups: declare supabase + cold-store services in the flake (code#1723) dezren39/darwin-nix#1042 extends modules/laptop-backups.nix in the repoArchive pattern. Defaults enable them under the host's existing services.laptop-backups.enable, so merge + a laptop switch activates them; the hand-written unit files are removed immediately after that switch (tracked here so it doesn't drift). Parse-verified on the laptop. New standing rule recorded: no new non-nix-backed services.
  6. github-actions
    assigned
    dezren39
    on Oct 28, 2026

  7. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    MemberAuthor

    Final state 2026-10-28 ~12:20 CDT — the laptop-service consolidation is code-complete:

    • amd: laptop services live in this flake — relocate modules from dezren39/nix, add repo-keeper + autocommit (code#1723) #1740 (merged): laptop-backups + codex-remote-control relocated from dezren39/nix into nixos/home/ (amd imports by path; zero inputs.dezren39-nix references remain); new laptop-repo-keeper (hourly) + laptop-autocommit (1-min tracked sweeps for openclaw/hatch/hatch-personal + daily untracked review).
    • Remove laptop modules (moved to the code flake, code#1723) dezren39/darwin-nix#1043 (merged): the modules + homeModules output removed from the Darwin flake.
    • repo-keeper: tracked-files only + timeout fix (code#1723) #1745 (merged): keeper fixes from its first live runs (timeout-vs-function bug; tracked-only commits).
    • Eval proof: the amd config evaluates with all 17 user timers, incl. supabase-backup, cold-store-nightly/drill (from the relocated module), laptop-repo-keeper, laptop-autocommit-{openclaw,hatch,hatch-personal}, laptop-untracked-review.
    • First runs (by hand, pre-switch): keeper pulled cairn +6 / nix +2 / opencode +2 / hatch +7 (ff), committed + pushed openclaw's drift; untracked review found nothing to classify. Report-only flags: 28 MiB tracked job log in hatch-personal; 35 MiB tracked codedb.snapshot in openclaw-personal.
    • Remaining: ONE step — on the laptop needs Drew's sudo (no passwordless sudo, no auto-upgrade unit on this machine). The command is in Drew's chat. Until then the hand-written supabase/cold-store timers are restored and firing on schedule (next supabase 16:42); after the switch they must be deleted (home-manager owns them) — the switch command Drew has includes that rm.

  8. dezren39 commented on Oct 28, 2026

    @dezren39
    dezren39
    on Oct 28, 2026
    MemberAuthor

    CLOSED OUT 2026-10-28 ~15:25 CDT — the consolidation is LIVE:

    • Switch completed on amd (generation xzwi99z…): the laptop now runs the flake with laptop-backups + codex-remote-control relocated in-repo, the supabase/cold-store units flake-owned, and the new keeper/autocommit/review timers active. First switch attempt failed on a single transient builder failure under system-units; the identical configuration built fully on retry with no code changes, and the second switch activated cleanly.
    • Verified live: user timers include laptop-repo-keeper (hourly), laptop-autocommit-{openclaw,hatch,hatch-personal} (every minute), laptop-untracked-review (daily), supabase-backup, cold-store-nightly/drill — unit files are home-manager store symlinks. The keeper service was started under systemd: Result=success, 23 primaries swept.
    • Keeper flags standing: ~/git/code primary parked on feat/agy-proxy-warm-stream-json with 8 dirty proxy files (left alone by design — a human/lane decision); a 'developing-today' primary on master with 9 dirty entries; theme-store diverged from its archived upstream; roc fetch exceeds the 60s timeout (degrades gracefully).
    • Deploy note: the switch ran from a clean worktree at origin/main (~/git/code-deploy) because the primary is parked; return deploys to the primary once it is back on main.
  9. dezren39
    added
    queue/queuedHas a queued task in the Supabase agent queue
    source/creatorFiled by the new-issue-creator job
    task/triageQueue packet kind: verify, reproduce, and route this item
    and removed
    meta/epicEpic / bundle of related work
    on Nov 1, 2026

  10. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    agent-note:v1
    state: triaged (coverage pass)
    Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
    Found via: coverage pass 2026-11-03

  11. dezren39
    added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    status/in-progressQueue worker lifecycle state
    and removed
    status/needs-triageNot yet triaged; needs a first look
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026

  12. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    agent-note:v1
    task: 7723
    lane: code-h
    stage: handed_off
    execution_lane: opencode

    Task 7723 (triage: #1723 — program to finish Tier-2 execution migration to the laptop) claimed by lane code-h at 2026-11-03T15:04Z; execution-lane assignment returned opencode with no session bound yet. Handing off to the opencode claimant per the worker runbook — this lane does not work the task directly. Claim labels synced on #1723 (work/assigned, queue/claimed, status/in-progress added; queue/queued and status/needs-triage removed).

  13. dezren39 commented on Nov 3, 2026

    @dezren39
    dezren39
    on Nov 3, 2026
    MemberAuthor

    Triage 2026-11-03 — program is live, narrowed to two Drew-gated remainders

    Verified on the laptop (execution host) at ~2026-11-03 11:00 CST, against fresh state: the laptop's own systemd user units, the queue health readout, and the hatch-personal cron registry snapshot (HEAD f41f9d725, 2026-11-02 16:25 CST — the sandbox is the client side of laptop-ssh.sh, so it cannot be inspected from here).

    Verdict: still live as a program tracker. No repo-side fix exists at current developing-today/code main — grep for pr-diff-review / issue-sweep-rotating in the repo returns nothing; those control jobs and this program's remaining work live in the hatch-personal cron registry and on the sandbox, not here. Do not close.

    Item status

    # Item State Evidence
    1 agent-queue-worker OPEN — the only live item Registry entry still enabled: true, every: 2h, at workspace/goals/supabase-muse-dev-platform-backups/crons/hourly/agent-queue-worker__interval@2h.md. It demonstrably still fires on the sandbox: memory/2026-11-02.md logs 05:43, 07:43 and 09:43 CST runs, each "claim_task returned all-null row — queue empty, no tasks worked this shift". Its source queue reads 0 queued / 16 done in today's queue health.
    2 cold-store nightly + drill DONE (re-verified live) cold-store-nightly.service last ran 2026-11-03 03:23 CST; cold-store-drill.service last ran 2026-11-01 05:21 CST, next 2026-12-01 (quarterly gate behaving). supabase-backup.service last ran 2026-11-03 04:45 CST. All three are linked home-manager store symlinks — flake-owned, per the no-non-nix-services rule.
    3 coord-mirror DONE by retirement No supabase-mirror or turso unit exists on the laptop (Drew's 2026-10-28 call: mirror retired, script kept in ~/tools/supabase-mirror/ for a rebuild).
    4 OpenChamber long tail DRAINED in practice Drew's 2026-11-02 rule ("no local repo copies on the sandbox; every task execs into the laptop") is in force, and this very task was executed by an OpenChamber scheduled lane on the laptop. No new sandbox-worktree sessions are being created; what remains is held leftovers (see below).

    Unlock condition — largely met, deletion not finished

    • Snapshot-first re-pointing: DONE and recorded as verified on 2026-11-02 (memory/2026-11-02.md): the git-hygiene runbook scoped laptop-only; all five new-issue-creator lanes read laptop primaries via laptop-ssh or gh api; stewardship-cycle and cold-store-eviction-scan moved to the laptop; ~/tools/fleet-fetch.py's local-clone machinery deleted (snapshots are gh-API only). The pr-diff-review job body now states "there are no sandbox clones (Drew, 2026-11-02)".
    • Clone deletion: mostly done. 20 sandbox ~/git copies were moved to recoverable trash (restorable until 2026-12-02) after pushing their unpushed work. 8 trees were held, not discarded: devtoday-opencode-embeddings (~10 dirty files), wt-model-by-id, devtoday-opencode + devtoday-code (the mains the held worktrees hang off), wt-4111-fix, wt-1150-sim-bin, fleet, nix-cache, pm-feed. Final removal of that held set is the last piece of the unlock step.

    Blockers and ordering

    1. (Drew's decision — item 1) agent-queue-worker needs a disposition: retire it, or migrate it laptop-side. Recommendation: retire. Its queue has been empty for days, and the repo-queue lane fleet (11 shift lanes in the same registry, plus scheduled OpenChamber lanes) now covers the work it was created to do. Vehicle either way is a hatch-personal cron-registry edit plus the sandbox-side remove/disable — neither is possible from the laptop.
    2. (Drew's go — unlock step) Resolve or accept the 8 held sandbox trees, then delete the remnants. Destructive, and the trash restore window to 2026-12-02 is the deadline that makes this worth doing soon rather than lapsing.

    Items 2, 3 and 4 need no further action; they can be ticked when someone next edits the Done-when list.

  14. dezren39
    added
    user/escalatedQueue or user escalation state
    and removed
    status/in-progressQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort/lLarge: multi-day or open-ended; expect staged check-inskind/infrastructureInfrastructure / host-portability workmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobtask/triageQueue packet kind: verify, reproduce, and route this itemuser/escalatedQueue or user escalation state

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions