[P1] atenet-router and dns run as single replicas with no PDB — every restart is a guaranteed full data-plane outage #645

Dedup note: verified against all open+closed issues 2026-07-30 — no issue
or PR files the data-plane SPOF. #181 ("Define substrate HA story") is
scoped entirely to ate-apiserver + its database and never mentions the
router or dns. PR #538 already did HA for ate-api-server (2 replicas +
round-robin), so a precedent exists. #617/#612/#614 are related but
distinct (see below).

Summary

All actor traffic flows through one atenet-router pod and all actor name
resolution through one dns pod:

• manifests/ate-install/atenet-router.yaml:110 — replicas: 1, no PDB,
  no topologySpreadConstraints.
• manifests/ate-install/atenet-dns.yaml:82 — replicas: 1, same.
• The router-managed Envoy Deployment is additionally hardcoded to one
  replica in code: cmd/atenet/internal/router/envoyrunner.go:140
  (replicas := int32(1)) — so this is a code change, not just a manifest
  bump.
• The only PDB in the repo belongs to ate-api-server
  (manifests/ate-install/ate-api-server.yaml:220).

Any voluntary or involuntary disruption — deploy, node upgrade, eviction,
OOM — takes the entire data plane down for its duration.

Evidence (live, GKE substrate-poc, 2026-07-30)

Measured blast radius with an in-cluster probe (5 req/s against a RUNNING
actor) while force-deleting the router pod:

• ~5.5 s of hard connection failures (curl code 000 — not graceful
  503s) in the best case: warm image, instant reschedule.
• During the 09:53Z GKE node-pool event the same failure mode produced
  6,880 dial tcp errors and 80 EOFs in the overnight soak — the full
  pod-reschedule + image-pull window.

Impact

• Guaranteed user-visible outage on every router deploy or node drain;
  failures surface as raw connection errors, which HTTP clients cannot
  safely retry for non-idempotent requests (POST).
• Defeats the "retries heal everything" design story: there is nothing to
  retry against while the single router is down.

Related but distinct (do not fold into this issue)

• [P3] XDS versionCount resets to 1 on router restart — config changes may not reach Envoy #617 — xDS versionCount resets on router restart (a config-staleness
  bug triggered by restart; also a hint that multi-replica xDS needs design
  attention, since each replica would keep an independent version counter).
• [P2] atenet controller reconcile short-circuits on Kubernetes API error — Envoy Deployment cannot self-heal #612 — atenet controller reconcile short-circuit prevents Envoy
  Deployment self-heal.
• [P2] No fault-injection e2e tests for any critical failure scenario #614 — lists "component crash + recovery (… atenet-router)" as an
  untested scenario; test-coverage gap, not the SPOF itself.
• Define substrate HA story #181 — HA umbrella; body covers only ate-apiserver and its store.

Suggested fix

1. Follow the PR Run ate-api-server in HA mode #538 pattern (which HA'd ate-api-server): ≥2 router
   replicas behind the Service, PDB maxUnavailable: 1, topology spread
   across nodes; same for dns. Make envoyrunner.go's replica count
   configurable.
2. Resolve the multi-replica xDS question first ([P3] XDS versionCount resets to 1 on router restart — config changes may not reach Envoy #617's version counter
   per-replica); if multi-replica is not yet safe, document the SPOF in the
   operations runbook (Need an operations runbook for Substrate operators #581) as an interim known limitation.