Repository navigation
Storage pools: RAID10 HDD tier for RustFS + striped/mirrored NVMe tiers with LINSTOR/DRBD #1813
Description
Relocated from dezren39/darwin-nix#504 on 2026-10-28 - it was filed in the wrong repo; developing-today/code is its home. Original content follows, verbatim.
Problem
Design and validate the cluster's storage before the hardware is committed. Drew's sketch (from the register — to be validated, not assumed): three tiers across 3 hosts, replicated with DRBD + LINSTOR so that data loss needs the same failure on 2 machines at once.
Source: idea register 2026-10-20 (goal stand-up-lab-cluster-talm-talos-cozystack-postgres-rustfs).
Drew's sketch
- HDD tier (cold-ish RustFS buckets): 3 hosts × 4 × 16 TB helium WD drives in RAID 10. (15–16 drives exist in total.)
- Fast NVMe tier: all Samsung 990 Pro.
- Mid NVMe tier: 980s, smaller 990s, good consumer SSDs — stripe 3 (≈ 990-class aggregate performance) × mirror 2 = 6 drives ("RAID 100": 3 stripes, 2 mirrors), so two drives must die to take down a machine.
- Two named pools from earlier work: (1) 4 drives RAID 10; (2) 6 drives RAID 100 (3 stripes × 2 mirrors).
- Goal: 3–5 devices with near parity in disk size and capability.
- Consumers: RustFS (S3) on the HDD tier; Postgres (CloudNativePG) and general volumes on NVMe tiers; LINSTOR/DRBD for cross-machine block replication.
Acceptance / first steps
- Map the validated hardware inventory (see inventory issue) onto the three tiers: exact drive counts per host, usable capacities per pool.
- Decide the layering: md RAID vs LINSTOR/DRBD-native striping/mirroring vs ZFS — and where each pool's redundancy actually lives (in-host vs cross-host), checking the "2 machines must fail" claim against the design.
- Benchmark the mid tier's 3-stripe claim against a single 990 Pro (does striping 980-class drives actually reach parity?).
- Define failure drills: pull a drive, pull a host; measure rebuild time/impact on 16 TB members.
- Emit the result as declarative per-node storage config feeding the TALM bootstrap issue.
Open questions
- NIC throughput between core hosts (pending inventory) — caps DRBD replication and rebuild speeds.
- RustFS erasure coding across hosts vs RAID 10 under it: one redundancy layer or two?
Acceptance criteria
- A storage design is recorded on this issue (or a linked design doc/PR) validating or revising the three-tier sketch (RAID10 HDD / fast NVMe / mid NVMe with DRBD + LINSTOR).
- The design states the replication/failure model: which single-machine failures are survived with no data loss.
- Drive counts are reconciled against the actual inventory (see Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810) before any purchase/commitment.
- Follow-up implementation issues are filed and linked for any build work.
Activity
developing-today commentedon Oct 28, 2026
CODE-747
dezren39 commentedon Oct 28, 2026
Comments carried over from dezren39/darwin-nix#504 (verbatim, in order):
dezren39 - 2026-10-26T21:32:51Z
Intake review
🤖 Fleet monitor intake pass (protocol §5).
Related: the 2026-10-20 lab-cluster idea register (linked in the body); depends on the hardware inventory in dezren39/darwin-nix#501; sibling issues dezren39/darwin-nix#502, dezren39/darwin-nix#503, dezren39/darwin-nix#505.
Live check: still open and unlabelled at intake; no merged PR resolves it — the sketch in the body is explicitly "to be validated, not assumed".
Notes / risks: tier sizes depend on the real drive inventory, which dezren39/darwin-nix#501 has not yet produced; validating the design before that lands risks rework.
Suggested approach: sequence after dezren39/darwin-nix#501: validate the tier/replication design against the inventoried drives, record the failure model, then file build follow-ups.
dezren39 - 2026-10-27T03:26:07Z
Superseded by #1085 (storage assembly & design validation epic)
The basement cluster program was seeded into its standard structure on 2026-10-26 (roadmap + epics + sub-issue tasks in developing-today/code, all in org Project #4 as Area=lab-cluster). Do not start work from this issue - work is tracked at #1085 (storage assembly & design validation epic). Full mapping is on nixdezren39/nix#508. Left open for Drew to close as superseded.
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
🤖 Agent note · done (triaged — live, blocked on hardware inventory, no code change possible yet)
effort/l; this is multi-day, not a triage-sized fix.status/needs-triageandqueue/queued; keptkind/infrastructure,effort/l,source/human,meta/enriched,priority/normal,task/triage.