EPIC: Basement Talos/Cozystack cluster program

Relocated from dezren39/darwin-nix#508 on 2026-10-28 - it was filed in the wrong repo; developing-today/code is its home. Original content follows, verbatim.

EPIC: Basement Talos/Cozystack cluster program

Turn the basement fleet — ~12 Threadripper machines, 15–16 × 16 TB helium HDDs, ≥3 TB DDR4 ECC — into a 3–5 node Talos Linux cluster under Cozystack, bootstrapped declaratively with TALM, plus the cheap-cloud program around it.

Source of truth: idea register 2026-10-20 (goal stand-up-lab-cluster-talm-talos-cozystack-postgres-rustfs). Research baseline: hidden_files/research-brief-2026-10-03.md in the goal (talm v0.35.0, Talos v1.14.1, Cozystack v1.6.4, RustFS 1.0 GA, managed Postgres = CloudNativePG).

Hard constraints (Drew, explicit)

No HashiCorp anything. Nomad was the plan; the company was sold. All-in on Kubernetes.
Everything bootstrappable and declarative. No undocumented hand-built state — that is why the machines have no OS right now. The fleet must be rebuildable from config in git.
Egress is the scarce resource, not compute: one 2 Gbps symmetric fiber link behind very fast, very cheap local hardware.
Cloud budget: Turso $5 / Cloudflare $5 Workers plans in play, free tiers preferred, $25–$30 possible later, $75 out.

Day-one cluster services

Postgres (CloudNativePG via Cozystack) · Cloudflare tunnel · Tailscale · NATS · RustFS (S3, cold tiers on HDD pools) · FluxCD running the app directly · DRBD + LINSTOR replication.

Child issues

Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810 — Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines (starts here: stats decide node order)
BMC/BIOS configuration audit + routine KVM screenshot verification system #1811 — BMC/BIOS configuration audit + routine KVM screenshot verification system
TALM bootstrap: declarative per-node disk config for non-identical drives #1812 — TALM bootstrap: declarative per-node disk config for non-identical drives
Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD #1813 — Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD
Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale #1814 — Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale
OCI always-free maximization: account setup + automation #1816 — OCI always-free maximization: account setup + automation
Research: boats.dev #1815 — Research: boats.dev

Suggested order

Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810 inventory (stats → node shortlist) and BMC/BIOS configuration audit + routine KVM screenshot verification system #1811 audit groundwork.
TALM bootstrap: declarative per-node disk config for non-identical drives #1812 TALM proof-of-flow on the first nodes, with Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD #1813 storage design feeding its per-node configs.
Cozystack + day-one services on the proven nodes.
Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale #1814 egress design before anything user-facing; OCI always-free maximization: account setup + automation #1816 OCI in parallel (blocked only on Drew's email/card choice).

Blockers / open questions for Drew (from the register)

Basement power capacity + mess: full-fleet power-on is a long tail; 3–5 nodes first.
Which app does FluxCD run first ("the ID app" — name/repo)?
OCI account email/card; exact NIC spec + switch gear; boats.dev context; Turso account state.

Acceptance criteria

Child issues Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810–Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale #1814 (and any later children) are tracked from this epic with their states kept current.
Node selection for the first 3–5 machines is made from Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810's hardware stats and recorded here.
The cluster bootstrap is reproducible from config in git (TALM + Talos + Cozystack), with no HashiCorp tooling anywhere in the path.
Day-one services from the register (Postgres/CloudNativePG, Cloudflare tunnel, Tailscale, NATS, RustFS, FluxCD, DRBD+LINSTOR) each have a child issue or a recorded deferral.

CODE-751

Comments carried over from dezren39/darwin-nix#508 (verbatim, in order):

dezren39 - 2026-10-26T19:40:20Z

🤖 Intake review — fleet monitor, 2026-10-26 run

Still live: this is the program epic — no merged PR found this run changes its state; children dezren39/darwin-nix#501–dezren39/darwin-nix#505 carry the first steps (per the body's child list, work starts at dezren39/darwin-nix#501 hardware stats, which decides node order).
Related: dezren39/darwin-nix#501 (hardware inventory), dezren39/darwin-nix#502 (BMC/BIOS audit), dezren39/darwin-nix#503 (TALM bootstrap), and the 2026-10-20 idea register linked in the body (source of truth, goal stand-up-lab-cluster-talm-talos-cozystack-postgres-rustfs). dezren39/darwin-nix#507 (OCI free tier) and dezren39/darwin-nix#506 (boats.dev research) are sibling program items filed the same day.
Notes / risks: hard constraints from the body bind every child: no HashiCorp tooling, everything declarative/rebuildable from git, egress (one 2 Gbps link) is the scarce resource, cloud budget $25–$30 possible / $75 out.
Suggested approach: keep this as the index only — execution lives in the children, in order, starting with dezren39/darwin-nix#501.

dezren39 - 2026-10-27T03:25:50Z

Program structure seeded in developing-today/code (2026-10-26)

This program now has its standard home, per Drew's project-structure direction:

Project home / roadmap: planning/roadmaps/basement-cluster.md in developing-today/code (PR docs(planning): seed basement cluster roadmap + first status record #1082, merged), registered in planning/README + root ROADMAP.
First status record: doc/2026-10-27T03-14-08Z_status_basement-cluster/ in developing-today/code.
12 subcomponent epics in developing-today/code, all in org Project Bump build/buck/preludes/buck2-prelude from 3a02ea3 to 9530656 #4 (Area=lab-cluster, Kind=Epic): epic(cluster): hardware inventory & first-node selection #1080 inventory, epic(cluster): firmware / BIOS / BMC audit & standardization #1081 firmware/BMC, epic(cluster): basement power & space #1083 power & space, epic(cluster): network fabric #1084 network, epic(cluster): storage assembly & design validation #1085 storage, epic(cluster): TALM + Talos bootstrap #1086 TALM/Talos, epic(cluster): Cozystack platform #1087 Cozystack, epic(cluster): platform services #1088 platform services, epic(cluster): observability, backup & recovery #1090 observability/recovery, epic(cluster): cloud extension (OCI free fleet & egress offload) #1091 cloud extension, epic(cluster): first workload - the ID app #1092 ID app, epic(cluster): ESP32 wing #1093 ESP32 wing.
Detailed tasks filed as sub-issues under the first three epics (code#1094-task(cluster): basement cleanup + rack layout for the first nodes #1106), labeled with the new fleet-wide mode/online|hybrid|physical family.
Dashboard: visualization/page work filed as impl(dashboard): project page + complexity-weighted progress visualization (seed: lab-cluster) #1108.

Mapping from this repo's Oct 20 issues: nixdezren39/nix#501 -> code#1080, nixdezren39/nix#502 -> code#1081, nixdezren39/nix#503 -> code#1086, nixdezren39/nix#504 -> code#1085, nixdezren39/nix#505 -> code#1091 (egress half), nixdezren39/nix#507 -> code#1091 (OCI half). nixdezren39/nix#506 (boats.dev research) is answered: no free tier, 7-day trial then $20/mo prepaid, egress uncharged (recorded in the goal register 2026-10-25).

Recommendation: close nixdezren39/nix#501-dezren39/darwin-nix#507 as superseded by the code epics above (this epic, nixdezren39/nix#508, can stay as the historical parent or close with them). Left open for Drew's call - linking comments posted on each.

dezren39 - 2026-10-28T20:30:47Z

Fleet scope correction (operator, 2026-10-28): this epic and its children (dezren39/darwin-nix#501–dezren39/darwin-nix#507) are the lab cluster program, tracked here since the 2026-10-20 idea-register filing. They are not nix-repo work. A bulk triage conversion on 2026-10-28 swept every open issue in this repo into the repo-health work queue indiscriminately, and fleet lanes worked dezren39/darwin-nix#506/dezren39/darwin-nix#507 and reported them as nix backlog items — that was the error. The five still-queued fleet triage tasks for dezren39/darwin-nix#501–dezren39/darwin-nix#505 are cancelled, the fleet-added escalation on dezren39/darwin-nix#506 is removed, and this batch is out of the repo-health fleet's scope. The items remain tracked here under this epic for the lab cluster goal; nothing about their content or priority within that program changes.

Relocation map (2026-10-28): this program's issues were filed into dezren39/nix on 2026-10-20 by mistake - they are developing-today/code items. GitHub transfer requires the same owner for both repos, so they were recreated here with bodies, labels, and all comments carried over verbatim, and the nix originals are closed as moved (pointers left on each).

Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines dezren39/darwin-nix#501 -> Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810 (hardware inventory)
BMC/BIOS configuration audit + routine KVM screenshot verification system dezren39/darwin-nix#502 -> BMC/BIOS configuration audit + routine KVM screenshot verification system #1811 (BMC/BIOS audit)
TALM bootstrap: declarative per-node disk config for non-identical drives dezren39/darwin-nix#503 -> TALM bootstrap: declarative per-node disk config for non-identical drives #1812 (TALM bootstrap)
Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD dezren39/darwin-nix#504 -> Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD #1813 (storage pools)
Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale dezren39/darwin-nix#505 -> Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale #1814 (egress strategy)
Research: boats.dev dezren39/darwin-nix#506 -> Research: boats.dev #1815 (boats.dev research)
OCI always-free maximization: account setup + automation dezren39/darwin-nix#507 -> OCI always-free maximization: account setup + automation #1816 (OCI always-free)
EPIC: Basement Talos/Cozystack cluster program dezren39/darwin-nix#508 -> EPIC: Basement Talos/Cozystack cluster program #1817 (this epic)

Sibling references in the bodies above were rewritten to the new numbers; references inside carried-over comments keep their original dezren39/nix#NNN form, since those comments are the historical record.

agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03

🤖 Agent note · done

Agent: queue worker (repo-worker-code), lane code-l, direct execution lane
Session / ID: task 7733
Found via: queue claim
Task: Triage of this epic (issue EPIC: Basement Talos/Cozystack cluster program #1817) at current state — verdict below.
Expectation: triage only; the epic itself stays open and is driven by its child issues.
Links: issue EPIC: Basement Talos/Cozystack cluster program #1817; children issue Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810, issue BMC/BIOS configuration audit + routine KVM screenshot verification system #1811, issue TALM bootstrap: declarative per-node disk config for non-identical drives #1812, issue Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD #1813, issue Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale #1814, issue Research: boats.dev #1815, issue OCI always-free maximization: account setup + automation #1816

Triage verdict (fresh gh check, 2026-11-03): still live — not handled, not superseded, no code fix path at current main.

This is a program epic for physical basement infrastructure (Talos/Cozystack cluster), not a repo change: there is nothing in this repository's main that implements or closes it, and its acceptance criteria are hardware/program outcomes. The fix path is the child-issue sequence, whose current states are:

issue Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810 — OPEN, queued for triage — hardware inventory/stats; starts the program (stats decide node order)
issue BMC/BIOS configuration audit + routine KVM screenshot verification system #1811 — OPEN, queued for triage — BMC/BIOS audit + KVM screenshot verification
issue TALM bootstrap: declarative per-node disk config for non-identical drives #1812 — OPEN, queued for triage — TALM bootstrap (after issue Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810)
issue Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD #1813 — OPEN, queued for triage — storage pools (feeds issue TALM bootstrap: declarative per-node disk config for non-identical drives #1812's per-node configs)
issue Egress strategy: 2 Gbps home fiber behind Cloudflare/Turso/Tailscale #1814 — OPEN, queued for triage — egress strategy (before anything user-facing)
issue Research: boats.dev #1815 — OPEN, carries user/escalated — boats.dev research; blocked on Drew's context for boats.dev
issue OCI always-free maximization: account setup + automation #1816 — CLOSED (not planned) — superseded: the OCI account work now lives on issue ops(cluster): OCI tenancy access + Always Free fleet provisioning (child of #1091) #4904 (linked under issue epic(cluster): cloud extension (OCI free fleet & egress offload) #1091, per the closing comment on issue OCI always-free maximization: account setup + automation #1816). Note: issue OCI always-free maximization: account setup + automation #1816 still carries a stale queue/queued label despite being closed.

Ordering recorded: issue #1810 + issue #1811 first → issue #1812 with issue #1813's storage design → Cozystack + day-one services on proven nodes → issue #1814 before anything user-facing.

Blockers that keep the epic from proceeding as code work: (1) the gating work is physical — basement hardware inventory/stats (issue #1810) and power capacity for the first 3–5 nodes; (2) open Drew questions already recorded in the epic body (which app FluxCD runs first, exact NIC/switch gear, boats.dev context on issue #1815). No new follow-up packet is enqueued here: every live child already carries its own queued task, and the epic's next action is those children's triage/execution, in the order above.

Task 7733 completes as triaged; this epic's queue labels are cleaned to match (no in-flight labels remain).

Navigation Menu

Repository navigation

EPIC: Basement Talos/Cozystack cluster program #1817