Local snapshot node pinning breaks worker fungibility and causes unrecoverable resume errors if the free workers are on a different node #589

Summary

Local snapshots are tied to the specific nodes where the actor was running when paused. However, workers that belong to the same worker pool can be spread across different nodes in the cluster. As a result, the hard dependency between paused actors (local snapshots) and specific nodes cannot always be satisfied when no workers are available on that node.

This issue is broader than autoscaling, but it completely breaks autoscaling operations. We need to remove or mitigate this hard node dependency when there are no workers available for that cluster/node.

More Details

1. Worker Pool Spread vs. Pinned Local Snapshots

   • In a multi-node cluster, pods in a WorkerPool are distributed across various nodes.
   • When an actor is paused with a local snapshot, its metadata records the specific node VM where the local snapshot resides
   • Because any subsequent resume requires a worker pod on that exact node, workers across the pool are no longer fungible for that actor. If all worker pods on that node are busy or none exist, the actor cannot resume—even if there are plenty of free workers on other nodes in the same worker pool.

2. Problems for Autoscaling

(see #198 (comment))

• Scale-Down Strands Paused Actors: When a worker pod or node is removed during scale-down (HPA, kubectl scale, etc), a paused actor's local snapshot remains pinned to a node that may no longer have any free worker pods (or any worker pods at all).
• Scale-Up Gives No Locality Guarantee: If paused actors with local snapshots cannot resume because no worker pods are available on their pinned node, scaling up the worker pool may not unblock them (if the new workers land on a different node)