Repository navigation
Machine pool: onboard additional machines as fleet job nodes and redesign placement around a pool #2179
Description
What this is
Drew, 2026-10-29: the fleet's job execution runs on effectively one machine (the laptop, amd-6 — OpenCode server, queue workers, OpenClaw gateway). He wants (1) additional machines set up so they can be used for jobs the same way the laptop is used now, and (2) the fleet structure redesigned to connect to a pool of machines instead of a single host.
Work
- Machine onboarding standard. Define what a machine needs to join the pool: NixOS config from the flake, SSH access pattern, OpenCode server (attach pattern), queue worker capability, health reporting, and the secrets it may hold (by name only). A machine joins by applying the standard, not by hand-crafting.
- Bring the additional machines online. The hardware exists (the Threadripper fleet in the basement, Osseo WI) — the on-site evidence cluster (code#1095/task(cluster): verify NIC and switch-gear facts #1097/task(cluster): collect current BIOS/BMC versions and settings - core machines #1099/task(cluster): bring core machines to the golden baseline #1101) is the inventory input and is Drew's physical step. As machines are inventoried and installed, onboard them one by one against the standard. Relationship to the lab-cluster goal (Talos/Cozystack): that program stands up the Kubernetes cluster; THIS issue is about machines as fleet job-execution nodes. Where a machine serves both, the standard says so explicitly — do not let the two programs silently diverge.
- Redesign the fleet structure for a pool. Today: hardcoded laptop assumptions (the 12-session cap, SSH target, scheduler residency, watchdog locality). Target: a machine registry (identity, capabilities, capacity, health), placement that picks a machine per job/session (the route-pool idea, one level down: machine pools beside model pools), and schedulers (Muse cron, OpenChamber scheduled tasks OpenCode scheduled tasks: migrate the Muse→OpenCode cron wrappers into OpenChamber scheduled tasks (waves, monitoring gates) #2174, OpenClaw automations OpenClaw scheduled tasks: use the native automations scheduler for watchdog-class and laptop-local work #2177) placing work across the pool. Observability follows the same move: per-machine attribution joins the per-job/per-model attribution in External-system observability: unified PM + queue + quota attribution (webhooks, hot list, session IDs, projections) #2072 and the Supabase mirror.
- Failure shape. A machine leaving the pool (off, overloaded, network) drains cleanly: its sessions/jobs are visible as orphaned in one place, and no other machine's state is corrupted (the plaza-objects/shared-state lessons).
Acceptance
- The onboarding standard written and applied to the first additional machine end-to-end: it runs a real queue job and an OpenCode session, reviewed like laptop work.
- The registry exists and the watchdog covers every pooled machine.
- A written placement design (which scheduler places what, on which signals) with the first placement rule live.
Related
Hardware evidence cluster code#1095/#1097/#1099/#1101 (Drew, on-site) · lab-cluster goal · #2072 observability · #2174 OpenChamber scheduled tasks · #2177 OpenClaw scheduled tasks · #2173 quota utilization (pool quotas are the other half of the throughput ceiling).
Activity
developing-today commentedon Oct 29, 2026
CODE-1072
dezren39 commentedon Oct 29, 2026
Intake review
kind/infrastructure,source/human(Drew directive, 2026-10-29),status/needs-triage,priority/high,effort/l,meta/enriched. The body already carries an## Acceptancesection, so no acceptance section was appended.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7308 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
5 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: handed-off
task: 7308
follow_up_task: 8927
worker: repo-worker-code
lane: aea12c82-c5af-4e5a-9b66-f8e360f87177
at: 2026-11-03T23:52:56Z
The triage task is complete; the first reviewable onboarding-standard and registry-shape step is queued as task 8927. Physical inventory and installation remain explicitly out of scope for that task.
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: picked_up
task: 8927 (small_fix, repo-health-code-pilot; worker repo-worker-code; lane code-d, execution lane direct)
by: repo-queue-worker lane D
Picked up task 8927: draft the machine onboarding standard and registry shape for issue #2179 (first reviewable, non-physical phase). Revalidated fresh: issue #2179 is OPEN with no hard-stop labels (no held / user/owned / do-not-merge / do-not-auto-edit) and no existing PR covers this doc. Working on the laptop in a dedicated worktree on branch queue/8927-2179-machine-onboarding-standard; deliverable is a PR referencing issue #2179 (the issue stays open — physical onboarding remains blocked on the inventory issues).
labels: work/assigned + queue/claimed + status/in-progress added; queue/queued removed.
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: done (pr_opened)
task: 8927 (small_fix, repo-health-code-pilot; worker repo-worker-code; lane code-d, execution lane direct)
by: repo-queue-worker lane D
Delivered: PR #5292 — #5292 — drafts the machine onboarding standard and registry shape for #2179 as
doc/2026-11-04T05-15-02Z_design_machine-onboarding-standard-v1/(branch queue/8927-2179-machine-onboarding-standard, commit f3177da, from origin/main ac14972; laptop worktree ~/git/fleet/queue-8927-2179-onboarding).Coverage against task 8927's acceptance: §1 onboarding contract (NixOS flake configuration via ./nixos/hosts, SSH access pattern, OpenCode server attach pattern, queue-worker capability, health reporting); §3 registry schema (identity / capabilities / capacity / health, registry identity distinct from the Talos inventory); §0 execution-node vs Talos/Cozystack boundary explicit; §1.6 secrets named by class only, no values; §5 separates software work that can proceed now from the physical blockers (#1095/#1097/#1099/#1101) and claims no onboarded node.
Issue #2179 stays open for the review round and the remaining acceptance (first-machine application, live registry, first placement rule), tracked in §6 of the doc. Labels set by this claim (work/assigned, queue/claimed, status/in-progress) are removed on completion per protocol.