Decide and document secrets-management/rotation strategy (K8s Secrets has no rotation story) #43

Context

docs/SECURITY.md already states, in its encryption-at-rest section, that there is no KMS/secrets-manager in v1 and names Vault (or equivalent) as the intended upgrade path — the same treatment given to other deliberately-deferred hardening items. Unlike those other items, this one has no tracking issue, which means it risks being silently forgotten rather than deliberately revisited, breaking the pattern every other "documented gap" in this repo follows (compare #26, #12, #33, all of which exist specifically so a documented gap has a home to graduate into real work from).

The gap

Today, secrets (Keycloak client secrets, DB credentials, MinIO keys) live in K8s Secrets created via Helm values / infra/k3s manifests, per CLAUDE.md's "Keycloak secrets currently hardcoded in app/core/config.py; move to K8s secrets before production (Phase 5)" note. K8s Secrets are base64-encoded, not encrypted, at the API object level (etcd encryption via --secrets-encryption covers the storage layer, per docs/SECURITY.md, but not secret rotation, access auditing, or centralized issuance). There's no rotation story, no audit trail of which secret was read by which pod/when, and no path for an admin to rotate a credential without editing YAML and redeploying. For a 10-year-lifecycle enterprise template, "how do we rotate the Postgres password without downtime" is a question that should have a documented (even if not yet automated) answer.

Scope

1. Decide and document the mechanism, the same deliberate way Move rate-limit state to a shared store when the backend scales past 1 replica #38 documents the rate-limit store decision and Backups: strategy, encryption, and retention reconciliation #12 documents the backup mechanism decision: Vault (as already named in docs/SECURITY.md) vs. a lighter-weight option (e.g. External Secrets Operator against a simpler backend, or SOPS-encrypted values files) vs. explicitly staying with K8s Secrets + etcd encryption and only adding rotation tooling around it. Write the decision and rationale into docs/SECURITY.md, replacing the current one-line "Vault is the upgrade path" note with the real design.
2. Rotation procedure: at minimum, a documented (ideally scripted) procedure for rotating each secret class (DB password, Keycloak client secret, MinIO keys) without full-stack downtime — this is the concrete gap docs/SECURITY.md doesn't currently answer at all.
3. If a secrets manager is chosen: an infra/ integration following the existing Helm chart patterns (no Bitnami, per CLAUDE.md's constraints — check the chosen tool's licensing/image provenance against the air-gap vendoring requirement in docs/AIRGAP_SETUP.md before adopting it).
4. UI element (required): an admin "Secrets" status screen (require_admin-gated, alongside Settings) showing, per secret class, its last-rotated timestamp and age (e.g. "DB password: rotated 47 days ago") — read-only status visibility, not an in-browser secret editor (secrets should never transit the frontend/browser). This requires the backend to track rotation timestamps somewhere (e.g. a small metadata table or annotation on the K8s Secret object, not the secret value itself) even before full rotation automation exists, so the screen has real data to show from day one.

Definition of Done

• docs/SECURITY.md's secrets-management section rewritten with a concrete decision (mechanism + rationale), not just "Vault is the upgrade path."
• A documented rotation procedure exists for each secret class currently in use (DB, Keycloak, MinIO).
• Rotation timestamps are tracked somewhere the backend can read (metadata table or Secret annotation) and exposed via a small backend endpoint.
• Admin-only "Secrets" status UI screen exists, shows rotation age per secret class, and is reachable from the app's normal navigation.
• If a secrets-manager tool is adopted: it is vendorable per docs/AIRGAP_SETUP.md and uses an official upstream image (no Bitnami), consistent with every other chart in infra/helm/charts/.
• Standard merge gate from AGENTS.md's Definition of Done applies in full (tests, lint, e2e for the new UI flow, docs).