[P1] WorkerPool scale-down and rolling updates crash running actors without checkpointing #621

Severity: P1 (data loss — actor in-memory state lost on every scale-down or rolling update)
Component: cmd/atecontroller/internal/controllers/workerpool_controller.go + workerpool_apply.go
Confirmed: Yes — code-level analysis

Summary

When a WorkerPool's replica count decreases or a rolling update replaces pods, Kubernetes
terminates worker pods that host running actors. There is no preStop lifecycle hook,
no automatic checkpoint/suspend of running actors before termination, and no ateapi
coordination before the SIGTERM is sent. After the grace period (default 300s), Kubernetes
sends SIGKILL, the actor sandbox is destroyed, and all in-memory actor state is lost.
The ateapi syncer eventually detects the pod deletion and marks the actor STATUS_CRASHED.

Impact

• Every WorkerPool scale-down or rolling update causes data loss for actors running
  on the terminated pods, regardless of whether the actor has external checkpoint config.
• Actors' in-memory state accumulated since the last checkpoint is lost.
• Users see STATUS_CRASHED actors after cluster operations they expect to be transparent.
• On a cluster with 100 running actors across 10 pods, a rolling restart (which Kubernetes
  performs 1 pod at a time by default) crashes ~10 actors per restart step.
• During upgrades, the blast radius is the entire pool.

Root Cause

File: cmd/atecontroller/internal/controllers/workerpool_apply.go lines 62, 149–159

podSpecAC.WithTerminationGracePeriodSeconds(int64(workerTerminationGracePeriodSeconds(wp)))
// Default: 300s

const defaultTerminationGracePeriodSeconds int32 = 300

The TerminationGracePeriodSeconds gives actors 5 minutes to complete in-flight
requests, but there is no preStop hook that triggers actor checkpointing.

File: cmd/atecontroller/internal/controllers/workerpool_controller.go lines 62–65

// Handle deletion
if !wp.GetDeletionTimestamp().IsZero() {
    log.Info("WorkerPool is being deleted")
    return ctrl.Result{}, nil  // No drain/checkpoint logic
}

The controller does not checkpoint actors before scaling down or deleting the pool.

The syncer marks the worker STATE_DRAINING when a pod gets a DeletionTimestamp
(so no NEW actors are scheduled to it), but existing RUNNING actors are not touched —
they continue until SIGKILL.

No preStop hook exists in the pod spec. A preStop hook that called
SuspendActor for each running actor on the pod would allow state preservation.

Steps to Reproduce

# 1. Create and resume actors with stateful counter increments
kubectl ate create atespace scale-test
for i in $(seq 1 5); do
  kubectl ate create actor actor-$i -a scale-test --template ate-demo-counter/counter
  kubectl ate resume actor actor-$i -a scale-test
done

# 2. Drive traffic to accumulate state
for i in $(seq 1 5); do
  for _ in $(seq 1 20); do
    curl -s -X POST -H "Host: actor-$i.scale-test.actors.resources.substrate.ate.dev" \
      http://localhost:8000
  done
done

# 3. Trigger a rolling update (change ateom image or any WorkerPool field)
kubectl patch workerpool counter -n ate-demo-counter \
  --type=merge -p '{"spec":{"replicas":2}}'

# 4. Observe actors crash
watch -n1 'kubectl ate list actors -a scale-test'
# Actors transition to STATUS_CRASHED

# 5. Attempt to resume a crashed actor and verify counter was reset to 0
kubectl ate resume actor actor-1 -a scale-test
curl -s -H "Host: actor-1.scale-test.actors.resources.substrate.ate.dev" http://localhost:8000
# Counter is 0 — all state lost

Expected Behavior

Before terminating a worker pod:

1. The controller (or a preStop hook) should call SuspendActor for each actor RUNNING
   on that pod.
2. Only after all actors on the pod are successfully suspended (or after a timeout with
   a graceful CRASHED fallback) should the pod proceed to SIGTERM.

Suggested Fix

Option A — preStop lifecycle hook: Add a preStop hook to the ateom container that:

1. Queries the local atelet for currently-running actor UIDs.
2. Calls SuspendActor for each via ateapi.
3. Waits for completion or the grace period, whichever comes first.

// In buildDeploymentApplyConfig:
containerAC.WithLifecycle(corev1ac.Lifecycle().
    WithPreStop(corev1ac.LifecycleHandler().
        WithExec(corev1ac.ExecAction().
            WithCommand("/ko-app/atelet-drain", "--timeout=240s"))))

Option B — controller-side drain: The WorkerPoolReconciler should detect replica
decreases and, before applying the new replica count, call SuspendActor for all
actors running on pods that will be terminated.

Option C — atelet SIGTERM handler: atelet intercepts SIGTERM and calls
SuspendActor for all actors it currently hosts before exiting.

Option A or C is preferred as they don't require the controller to know which specific
pods are being terminated by Kubernetes.