Skip to content

Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810

Open

Description

@dezren39

Relocated from dezren39/darwin-nix#501 on 2026-10-28 - it was filed in the wrong repo; developing-today/code is its home. Original content follows, verbatim.


Problem

We have ~12 basement Threadripper machines and no per-machine records. We cannot choose the first 3–5 Talos/Cozystack cluster nodes — or keep them at parity — until we know exactly what each box contains. Drew's directive: gather all hardware stats first, so the cluster starts with the best / most-identical machines.

Source: idea register 2026-10-20 (goal stand-up-lab-cluster-talm-talos-cozystack-postgres-rustfs).

What we know

  • Core machines are the same motherboard, same CPU, same specs except SSDs; ≥3 TB DDR4 server ECC RAM across the fleet.
  • 15–16 × 16 TB helium WD server-pull HDDs; every machine has some NVMe boot drive (best: Samsung 990 Pro); fast NVMe tier is "all 990s", mid tier mixes 980s / smaller 990s / good consumer SSDs.
  • Core machines have enterprise BMC/IPMI ("ASM") with KVM; NIC spec on core machines is unconfirmed ("at least 200 Gb ports" as dictated — exact spec TBD here).
  • GPUs are not a resource (3× Intel B50, 1× RTX 2060, old display-only cards) — record them, don't plan around them.
  • Blockers: basement power capacity and mess — full-fleet power-on is a long tail, so stats gathering must work machine-by-machine, including via BMC without a full boot where possible.

Acceptance / first steps

  • Define the per-machine record schema: motherboard, CPU, RAM (total + DIMM layout), every SSD/HDD (model, size, firmware, SMART/health, wear), NICs (model, speed, ports), BMC/KVM IP, BIOS version, BMC firmware version, PSU(s), GPU.
  • Gather stats for all ~12 machines (BMC/Redfish/iKVM where possible; minimal boot media otherwise) and store the records in this repo (or the goal's files/) in a diffable text format.
  • Produce a ranked shortlist of the first 3–5 cluster nodes: parity of disk size/capability first (3 hosts × 4× 16 TB for the HDD tier; matching NVMe tiers), then health.
  • Pin down the exact NIC spec and the switch gear between core machines (feeds the LINSTOR/DRBD design issue).

Open questions

  • Exact NIC spec / switch hardware (register open question 5).
  • Are all 15–16 HDDs healthy enough for RAID 10 pools, or are some spares-only?

Acceptance criteria

  • A per-machine hardware inventory exists (linked from this issue) for the ~12 basement machines: CPU, motherboard, RAM, and especially SSD/drive details per box.
  • The first 3–5 cluster machines are picked from that inventory, with the parity reasoning recorded.
  • The inventory is in a form that can be re-gathered/refreshed, not a one-off prose list.

Activity

  1. added
    source/humanFiled directly by a human
    priority/normalOrdinary priority (the default)
    effort/lLarge: multi-day or open-ended; expect staged check-ins
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    kind/infrastructureInfrastructure / host-portability work
    on Oct 28, 2026
  2. developing-today commented on Oct 28, 2026

    @developing-today
    Contributor
  3. dezren39 commented on Oct 28, 2026

    @dezren39
    MemberAuthor

    Comments carried over from dezren39/darwin-nix#501 (verbatim, in order):


    dezren39 - 2026-10-26T21:33:52Z

    Intake review

    🤖 Fleet monitor intake pass (protocol §5).

    Related: the 2026-10-20 lab-cluster idea register (linked in the body); this inventory gates dezren39/darwin-nix#503 (TALM per-node config) and dezren39/darwin-nix#504 (storage tiers); sibling issues dezren39/darwin-nix#502, dezren39/darwin-nix#505.

    Live check: still open and unlabelled at intake; no merged PR resolves it — the body states there are no per-machine records yet.

    Notes / risks: machines are stated identical except SSDs, so drive details are the discriminating data; an inventory that skims drives will not support the node pick.

    Suggested approach: gather stats first (as Drew directed in the body), publish the inventory, then record the 3–5 node pick — several sibling cluster issues sequence behind this one, so it is the highest-leverage of the set.


    dezren39 - 2026-10-27T03:25:59Z

    Superseded by #1080 (hardware inventory & first-node selection epic)

    The basement cluster program was seeded into its standard structure on 2026-10-26 (roadmap + epics + sub-issue tasks in developing-today/code, all in org Project #4 as Area=lab-cluster). Do not start work from this issue - work is tracked at #1080 (hardware inventory & first-node selection epic). Full mapping is on nixdezren39/nix#508. Left open for Drew to close as superseded.

  4. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    on Nov 3, 2026
  5. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    agent-note:v1
    state: triaged (coverage pass)
    Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
    Found via: coverage pass 2026-11-03

  6. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026
  7. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

    • Agent: repo-worker-code (lane code-k)
    • Session / ID: queue task 7743
    • Found via: repo-health-code-pilot atomic claim
    • Task: Triage Hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines #1810 (hardware inventory + hardware-stats gathering: pick the first 3–5 cluster machines).
    • Fresh state: Issue OPEN; no hard-stop labels (held / user/owned / do-not-merge / do-not-auto-edit absent). Claim labels synced by this lane.
    • Handoff: Execution-lane assignment for task 7743 is opencode; the laptop claimant launches the bound opencode session, which drives the triage to its terminal state. This lane does not work the task directly.
  8. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    agent-note:v1 — task 7743 handed off (Lane A worker, 2026-11-03): execution lane assigned opencode; the claimant launches the bound session to drive task 7743 (triage of this issue) to its terminal state. This lane does not work it directly.

  9. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

  10. added
    queue/queuedHas a queued task in the Supabase agent queue
    and removed
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 3, 2026
  11. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · handed off

    • Agent: fleet monitor (repo-fleet-monitor)
    • Session / ID: task 7743
    • Found via: monitor reconciliation
    • Task: Claim-eviction repair: the queue task for this issue is queued and unclaimed (its previous claim's lease expired without a complete/park/block), but the issue labels still said claimed. Labels reset to the truth: queue/queued + status/needs-triage. Checkpoint stage on the task: handed_off.
    • Expectation: a queue lane re-claims the task and resumes from the checkpoint stage named in the task progress.
  12. removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 3, 2026
  13. dezren39 commented on Nov 3, 2026

    @dezren39
    MemberAuthor

    agent-note:v1
    state: picked-up → done (superseded)
    queue: repo-health-code-pilot
    task: 7743 (done, outcome superseded)
    by: repo-worker-code (openchamber scheduled lane c8b9355e-319a-4640-8096-7ad19480d151)
    at: 2026-11-03T21:05Z

    Revalidated issue #1810 as OPEN. Its existing 2026-10-27 note explicitly says it is superseded by #1080 and instructs not to start work here. #1080 is still OPEN and its latest triage identifies the genuine physical-access blocker; task 7624 is already blocked/escalated for that decision. No relevant PR supersedes or resolves this work. Task 7743 completed as superseded; #1810 is now an informational pointer to #1080, not an independent work item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort/lLarge: multi-day or open-ended; expect staged check-inskind/infrastructureInfrastructure / host-portability workmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)source/humanFiled directly by a humantask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions