[P2] atenet controller reconcile short-circuits on Kubernetes API error — Envoy Deployment cannot self-heal #612

Severity: P2 (Envoy Deployment drift not corrected during K8s API flap)
Component: Network Plane — cmd/atenet/internal/router/controller.go
Audit ID: NET-5

Summary

The atenet router's reconcile() function calls readyTemplates() first and returns
early if it fails. readyTemplates() performs a K8s API List across all namespaces —
an expensive, frequently-failing call during API server stress. The XDS snapshot and
the Envoy Deployment reconciliation (envoyRunner.reconcile()) do not depend on
template data, but both are gated behind this check. A Kubernetes API flap prevents
Envoy Deployment self-healing: if the Deployment is manually scaled to 0 during the
flap, it stays at 0 until the API recovers.

Root Cause

File: cmd/atenet/internal/router/controller.go lines 88–99

func (c *Controller) reconcile(ctx context.Context) error {
    templates, err := c.atStore.readyTemplates(ctx)
    if err != nil {
        c.logger.Error("Failed to get ActorTemplates", "err", err)
        return err  // ← short-circuit: UpdateSnapshot and envoyRunner.reconcile NEVER RUN
    }
    if err := c.xdsServer.UpdateSnapshot(ctx, templates); err != nil { ... }
    if err := c.envoyRunner.reconcile(ctx); err != nil { ... }
    return nil
}

The XDS snapshot update (UpdateSnapshot) uses only templates for route rules, but
an empty template list is a valid (safe) input — it just means no routes are advertised.
envoyRunner.reconcile() manages the Envoy Deployment and Service and does not use
templates at all.

Steps to Reproduce

1. Simulate a Kubernetes API server flap:

   # On kind: temporarily stop the API server container
   docker pause <kind-control-plane-container>
   sleep 30  # API unreachable for 30s
   docker unpause <kind-control-plane-container>

2. During the flap, manually scale the Envoy Deployment to 0:

   # Use raw kubectl before docker pause, queue the command
   kubectl scale deployment envoy -n ate-system --replicas=0

3. After the API recovers, observe:

   kubectl get deployment envoy -n ate-system
   # READY: 0/0  ← not recovered to 1/1

4. Check controller logs — reconcile() was returning early on every cycle during
   the flap and never called envoyRunner.reconcile() after the API recovered. The
   next successful reconcile (which includes readyTemplates) does fix it, but there's
   a window where Envoy is down and all actor requests fail.

Expected Behavior

UpdateSnapshot and envoyRunner.reconcile() should not be gated on readyTemplates.
An empty template list is a safe input. A K8s API flap should never prevent Envoy
Deployment self-healing.

Suggested Fix

Split reconcile into two independent paths:

func (c *Controller) reconcile(ctx context.Context) error {
    // Always runs — does not depend on templates
    if err := c.envoyRunner.reconcile(ctx); err != nil {
        c.logger.Error("Failed to reconcile Envoy", "err", err)
    }

    // Best-effort — failure does not block Envoy reconciliation
    templates, err := c.atStore.readyTemplates(ctx)
    if err != nil {
        c.logger.Warn("Failed to get ActorTemplates — XDS snapshot not updated", "err", err)
        return err
    }
    return c.xdsServer.UpdateSnapshot(ctx, templates)
}