Repository navigation
TALM bootstrap: declarative per-node disk config for non-identical drives #1812
Description
Relocated from dezren39/darwin-nix#503 on 2026-10-28 - it was filed in the wrong repo; developing-today/code is its home. Original content follows, verbatim.
Problem
Drew's open design question, verbatim from the register: how do you declaratively configure, in Talos/TALM, a bunch of hard drives that are all different per machine?
The cluster must be bootstrapped with TALM (cozystack/talm) — turn on bare machines running Talos, connect them, and TALM harvests each node's actual hardware into named per-node configs that we keep in git. Drew got this working locally with a couple of containers, barely; the full bare-metal flow is unproven. Hard constraints: everything bootstrappable/declarative, no HashiCorp anything.
Source: idea register 2026-10-20. Research baseline: hidden_files/research-brief-2026-10-03.md in the goal (talm v0.35.0, Talos v1.14.1, Cozystack v1.6.4).
What we know
- Target: 3–5 machines (chosen by the hardware-inventory issue) as the first Talos/Cozystack nodes.
- Host OS approach: Talos's small, frozen read-only host image on each machine's NVMe boot drive; the rest of the NVMe is for minor logs and wear-leveling headroom across consumer SSDs (best: 990 Pro).
- Per-node drives differ (SSD models/sizes vary; HDD counts vary) — the per-node config harvest is presumably the answer; document exactly how disks get matched/named per node.
- Downstream consumers of this config: the storage-pools issue (RAID/LINSTOR layouts) and Cozystack services (Postgres, NATS, RustFS, FluxCD, Tailscale, Cloudflare tunnel).
Acceptance / first steps
- Research talm's discovery/harvest flow for heterogeneous disks: how drives are identified in per-node configs, what happens on disk replacement.
- Prove the flow end-to-end on the first nodes: bare machine → Talos booted → talm harvest → named per-node config committed to git → reproducible re-apply.
- Document the frozen read-only host-image layout (OS partition vs logs/wear space) for our NVMe boot drives.
- Write the runbook: adding a node, replacing a disk, wiping/rebuilding a node from config only.
Open questions
- How are non-identical system disks handled if a boot NVMe dies and is replaced with a different model/size?
- Where do the harvested per-node configs live — this repo, or a dedicated cluster-config repo?
Acceptance criteria
- A worked TALM/Talos approach for per-node declarative disk config on non-identical drives is recorded on this issue (or a linked doc/PR), proven on at least the container/local flow Drew already has partially working.
- The approach keeps everything bootstrappable/declarative and uses no HashiCorp tooling (hard constraint in the body).
- Follow-up issues are filed and linked for the bare-metal proof on the first cluster nodes (see Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810).
Activity
developing-today commentedon Oct 28, 2026
CODE-746
dezren39 commentedon Oct 28, 2026
Comments carried over from dezren39/darwin-nix#503 (verbatim, in order):
dezren39 - 2026-10-26T21:33:04Z
Intake review
🤖 Fleet monitor intake pass (protocol §5).
Related: the 2026-10-20 lab-cluster idea register (linked in the body); sibling issues dezren39/darwin-nix#501 (inventory / first nodes), dezren39/darwin-nix#502, dezren39/darwin-nix#504, dezren39/darwin-nix#505.
Live check: still open and unlabelled at intake; no merged PR resolves it — the full bare-metal TALM flow is described in the body as unproven.
Notes / risks: per-node drive differences are the core difficulty; a scheme that only works on identical disks does not meet the stated problem.
Suggested approach: prove the harvest-to-named-config flow on the local/container setup first and record it, then schedule the bare-metal proof once dezren39/darwin-nix#501 picks the first nodes.
dezren39 - 2026-10-27T03:26:05Z
Superseded by #1086 (TALM + Talos bootstrap epic)
The basement cluster program was seeded into its standard structure on 2026-10-26 (roadmap + epics + sub-issue tasks in developing-today/code, all in org Project #4 as Area=lab-cluster). Do not start work from this issue - work is tracked at #1086 (TALM + Talos bootstrap epic). Full mapping is on nixdezren39/nix#508. Left open for Drew to close as superseded.
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
3 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: picked-up
task: 7746 (repo-health-code-pilot, triage)
lane: 8ac61616-3815-4e7b-b33d-840a3829154a
worker: repo-worker-code
Triage pickup. Verifying live state at current main, then scoping fix path or closing with evidence.
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged
task: 7746 (repo-health-code-pilot, triage) — lane 8ac61616-3815-4e7b-b33d-840a3829154a
verdict: live, not duplicate-by-closure — this is the missing concrete child of epic #1086
Live check at current main (
d3476bce, 2026-11-03)infra/talos/01b.us/is explicitly marked historical, torn down (v1.9-era test cluster).planning/roadmaps/basement-cluster.mdlists "TALM and Talos bootstrap" as planned, owned by epic epic(cluster): TALM + Talos bootstrap #1086.#503indezren39/nix(this issue's origin) is correctly closed there; the content is fully carried over.Answer to the design question (verified against current TALM/Talos docs)
Mechanism: one node file per node, plus a per-node body that is a strategic merge patch.
talm template -iagainst a node booted in maintenance mode writesnodes/<hostname>.yaml: a# talm: nodes=[...], endpoints=[...]modeline, the discovered hardware as comments (each disk's serial, wwid, model, size, transport), and a filledmachine.install.disk.talm apply. Fields outside networking (machine.install.disk,machine.install.diskSelector, extra etcd args) merge as expected. This is the declarative per-node disk hook — the cluster template stays identical for all machines; onlynodes/<node>.yamldiffers.talm applyrefuses a multi-node modeline with a body). That is the desired property here.Disk matching rule for non-identical drives:
machine.install.diskSelectorwithserialorwwid, notsize/model.serial/wwidstay unique per disk;size: '>= 1TB'ormodel: 'WDC*'can silently match a different disk when a node is reinstalled after a drive swap.UserVolumeConfig.spec.provisioning.diskSelector.matchtakes a CEL expression overdisk.{serial,wwid,model,size,transport,symlinks}, with the!system_diskguard so a volume can never land on the boot NVMe even if its selector is loose.machine.disks[]is deprecated in v1.10 — do not build on it.talm template -i→ hand-edit only the disk fields →talm apply --dry-run→ apply. That is reproducible with no hardware and can be proven first.Boot-disk replacement with a different model/size:
diskSelectoris evaluated at install time only, and a volume config is applied only while the volume is unprovisioned. So a swap is: editnodes/<node>.yaml(new serial/wwid, or new size/model if the selector was size-based),talm template -Ito refresh harvested comments,talm apply --dry-runto confirm the disk resolves to the intended device, then reinstall + re-apply. This is exactly the "wipe and rebuild from config alone" drill #1086 already requires, so it should be written as one runbook step rather than a separate mechanism.Where harvested configs live: recommend
infra/cluster/<cluster>/in this repo — node files change with PR review, sit next to the runbook and the roadmap, and avoid a second repo whose access model would have to be re-established. Not a blocker; flag for Drew only if a separate cluster-config repo is preferred.Blockers and ordering
Ordering: docs answer here → local container proof → bare-metal proof under #1086. The bare-metal half must not be attempted before #1810 lands, but the docs half should not wait for it.
Follow-up filed
Queue task enqueued for the docs deliverable (worked TALM/Talos per-node disk answer + add-node / replace-disk / rebuild runbook, committed under
infra/cluster/with the container-flow proof notes). It is scoped to be doable offline and lands as a PR that closes this issue's acceptance bullet 1–4.Found via: coverage pass 2026-11-03, triage task 7746.
developing-today commentedon Nov 3, 2026
🤖 Agent note · picked up
infra/cluster/, with parse-checked YAML examples (node file with serial diskSelector; UserVolumeConfig CEL with!system_disk) and the local container-flow proof notes. Bare-metal proof stays gated on Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810, out of scope.queue/8148-1812-talm-disk-runbookdocs(infra/cluster): TALM/Talos per-node disk config answer + runbook (…
developing-today commentedon Nov 3, 2026
🤖 Agent note · done
d5fa8062bfe219a75eaae5e018cfde988378694b) addsinfra/cluster/README.md+infra/cluster/examples/— the TALM node-file mechanism, identity-based disk matching (diskSelectorserial/wwid;UserVolumeConfigCEL with!system_disk), boot-disk layout, the add-node / replace-disk / rebuild-from-config runbook, and the local container-flow proof notes, all transcribed from the verified triage record (task 7746) with sources cited. Both YAML examples parse (verified; laptop copies md5-identical to the parsed copies).infra/cluster/<cluster>/here; a separate repo would be Drew's call).