Phase 5: K3s deployment walkthrough + encryption-at-rest verification #9

Context

Blocked by #8 (charts must exist first).

The repo owner has never deployed a Kubernetes cluster. This issue covers standing up K3s and deploying EAIStack to it for the first time, and — critically — is where every Phase 5 compliance control gets proven rather than merely implemented.

Why this matters beyond "get it running"

Phase 5's TLS and encryption-at-rest work ships implemented but unverified. CI validates rendered YAML; it cannot prove a TLS handshake succeeds, that etcd is actually encrypting, or that a volume is genuinely on an encrypted device. Until this issue is done, the compliance posture is a claim, not evidence.

Assume zero prior K8s knowledge throughout (CLAUDE.md states this explicitly).

Scope

1. infra/k3s/README.md — replace the stub:
   • K3s install with --secrets-encryption, explained flag by flag
   • Node-layer LUKS setup (cryptsetup, /etc/crypttab) — this is what at-rest encryption actually depends on; a StorageClass alone encrypts nothing
   • Explicitly: --secrets-encryption protects Secrets in etcd and does not encrypt PVC data. These are different mechanisms and the distinction is routinely missed.
2. infra/k3s/storageclass-encrypted-example.yaml — a working example.
   ⚠️ The one currently in docs/SECURITY.md is invalid: provisioner: kubernetes.io/local with parameters: {encryption: luks} — that parameter is silently ignored, yielding an unencrypted volume that looks configured. Worst possible failure mode for a compliance control. Replace it; do not copy it.
3. infra/scripts/verify-encryption-at-rest.sh — auditor-facing evidence:
   • Secrets in etcd are ciphertext, not plaintext
   • Every PVC bound to the approved encrypted StorageClass
   • Node-level crypt device check (lsblk, cryptsetup status) — the only one of the three that proves effect rather than configuration
4. Walkthrough doc — ordered, verifiable steps: install K3s → LUKS → cert-manager → kubectl apply -f infra/tls/ → helm install → verify. Each step needs a "how do I know this worked?" check.

Live-cluster verification (the whole point)

• kubectl get certificate -n eaistack — all Ready
• Postgres: SELECT ssl FROM pg_stat_ssl JOIN pg_stat_activity USING (pid) WHERE pid = pg_backend_pid(); → t
• Negative control: sslmode=disable must be rejected. TLS clients can opt out of is not a compliance control.
• MinIO: curl --cacert over https succeeds, plain http does not
• Retention CronJob actually schedules and runs

Expected friction

Per the plan, first-deploy problems concentrate in cert SANs and CA-bundle mounting — the two things unit tests structurally cannot cover. Budget time there.

Done when

A person who has never used Kubernetes can follow the doc start to finish, and verify-encryption-at-rest.sh produces evidence output suitable for an auditor.