Handle Pod Eviction/Deletion gracefully #23

Today Substrate doesn't have any special handling of worker pod deletions. If the worker pod assigned to a RUNNING actor is deleted, the system gets into a corrupted state where the control plane still thinks it's RUNNING. Moreover, we don't do any graceful suspension of the actor.

Additionally, the way we sync worker pods into the control plane Valkey is fragile and can drift.

This issue is to design a more robust handling of pod deletion, where we hook into the deletion process (via a finalizer) and perform a graceful worker pod deletion. This involves suspending the actor. It's unclear to me if it's needed to also resume it on a different worker pod, sort of a live actor migration, but I'll assume that just suspending is enough for now.

The model I have in mind is one where we externalize the registration and de-registration of worker pods to a dedicated controller running centrally inside atecontroller. I'll refer to this controller as the "wp-controller".

Worker Pod Creation

• The user creates or updates a WorkerPool custom resource.
• The WorkerPool controller creates a Kubernetes Deployment. Pods get a finalizer (e.g., ate.gke.io/worker-cleanup).
• The wp-controller observes the newly created Pod object and immediately calls RegisterWorker(NOT_READY) on ate-apiserver.

Registration

• Kubernetes schedules the worker pod, assigns it an IP address, and starts the ateom container.
• Kubelet periodically probes worker pod readiness. Once ateom completes boot and verifies internal runtime health, the probe passes and kubelet marks the Pod condition as Ready = True.
• The wp-controller observes the Ready = True Pod update and calls MarkWorkerReady(IP) on ate-apiserver to transition the worker to READY.

Running

• The worker is now in FREE state.
• When an actor needs to be resumed, the ateapi server picks this worker and assigns the actor to it.
• The worker handles requests for that actor until the actor is suspended.

Deregistration / Deletion (Controller-Driven)

• Kubernetes initiates a Pod deletion (e.g., scale down, node drain), setting deletionTimestamp on the Pod and sending SIGTERM to the ateom container (we will need to set a generous grace period on the worker pod).
• ateom intercepts SIGTERM and sheds incoming HTTP traffic by returning 503 Retry-After.
• The wp-controller observes the deletionTimestamp != nil update on the Pod and drives the graceful suspension:
  1. Calls CordonWorker on ate-apiserver to prevent new actor assignments.
  2. Checks if the worker has a running actor. If so, calls SuspendActor on ate-apiserver. ate-apiserver then calls atelet over gRPC to initiate a memory checkpoint of the active actor.
  3. Once suspension completes (or if no actor was assigned), calls UnregisterWorker on ate-apiserver to purge the worker from Valkey.
  4. Removes the finalizer from the Pod, allowing Kubernetes to permanently purge the Pod object from etcd.