Repository navigation
Prefer workers with cached actor images during scheduling #276
Description
Description
Atelet maintains a node-local image cache, but the scheduler does not know which images are cached on each node. As a result, an actor may be assigned to a cold node and pull its images again, even when another eligible node already has them cached.
Image preparation happens synchronously during ResumeActor, so unnecessary pulls increase actor startup latency and registry traffic.
Proposal
- Let atelet report its cached image digests to the control plane.
- Store cache information by node.
- After applying existing constraints, prefer available workers whose node has all required actor images cached.
- Fall back to the current random selection when no cache hit is available.
Local snapshot availability remains a hard constraint. Image cache affinity is applied only among nodes that can restore the selected snapshot.
A general scoring framework is not required: use cache-hit preference with fallback.
Expected Outcome
Reduce unnecessary image pulls and improve Actor Resume latency and consistency.
Related Issues
- Avoid re-untarring actor image rootfs on every restore #166 and Use a node-local overlayfs rootfs cache to eliminate per-restore untar during actor resume #228 optimize node-local image/rootfs reuse.
- RFC: Flexible & Non-Ossifying Control Plane Persistence Layer #135 discusses locality-aware scheduling for cached snapshots.
- Allow large image pulls to succeed by decoupling workflow deadline from lock TTL #233 tracks cold image pulls exceeding the
ResumeActordeadline.
This proposal complements those efforts by making worker selection aware of node-local image cache availability.
Activity
han2ni3bal-pixel commentedon Jun 26, 2026
Following up on Julian Gutierrez Oschmann (@juli4n)'s comment on PR #298 — I agree we should agree on a unified approach for node-locality before landing more one-off scheduling features.
Where we are today
Only one node-locality mechanism is merged: when an actor is paused with a local snapshot, NodeVmsWithLocalSnapshots on the Actor record is a hard constraint during ResumeActor — workers must be on one of those nodes.
#276 / PR #298 proposes a second path (not merged yet): atelet periodically reports OCI image digests, control plane stores them per node with a TTL, and the scheduler prefers workers whose node already has the required images, falling back to random selection.
The concern isn't that we already run two models in production — it's that merging #298 as a standalone RPC and store key sets a pattern that #228 (rootfs overlay), #135 (hot snapshot cache), and future volume locality will likely copy unless we align now.
Proposed direction
Split two concerns:
Keep the Prefer workers with cached actor images during scheduling #276 scheduling model: fixed layers (hard filter → ordered soft preferences → random fallback), not a general scoring framework.
Soft preference order (when we add them): hot snapshot on node first, then rootfs overlay (#228), then image tar cache (#276). Pause local snapshots remain hard-only via the Actor record — don't move them into node inventory.
Rough plan
Phase 0 (this issue): agree on hard vs soft split, node inventory as the long-term envelope, and preference ordering. Don't ossify ReportNodeImageCache as the permanent API.
Phase 1: implement #276 on NodeInventory directly (rebase PR #298 onto that, rather than merge then rename).
Phase 2: add rootfs overlay digests from #228.
Phase 3: add resident snapshot prefixes for Suspend hot-restore (#135 light).
Phase 4 (later): secondary indexes / #135 ListOptions only if query patterns prove we need them.
Happy to help turn Phase 0 into a short design note here if that's useful.
vicentefb commentedon Oct 30, 2026
adding a measurement that argues for this.
I ran an RL trainer shape on Substrate: 32 images with 16 rollouts each per step, 60 steps, 1,920 distinct images, 783 nodes. Every rollout of an image landed on a different node, and every node pulled the image.
One more thing: a restore from a golden snapshot onto a node without the image costs the same as booting the image, because the golden download and the gVisor restore together are about 0.3 s. So the preference you describe is also what makes golden snapshots pay off.
tomergee commentedon Nov 2, 2026
Another consumer, and two asks for the unified node-locality approach this thread wants.
In ai-on-gke/sandboxenv-rl we are planning a pre-flight that fills node image caches before an RL job (ai-on-gke/sandboxenv-rl#205): with placement image-blind, the first boot of each image pays the unpack on whatever node it lands on, and #3481's numbers (unpack 1 ms with the layers on the node, 9–50 s without, ~97 % of a cold first command) are what the job sees. The workaround available on main is client-side: WorkerPools as placement domains, each with its own label, and
Actor.worker_selectorper actor to keep an image's actors inside the domain that holds it. A scheduler preference like PR #298 would retire that.Until it lands, a client needs two things the API does not give today:
kubernetes.io/hostname, zone, node pool) intoWorker.labelsat registration, whichworker_selectorcan then name, or a per-actor node affinity composed into theRequiredNodesthe scheduler already has for local snapshots.Worker.status) lets a client verify a warm node and choose where to send an image's actors, and it is also how an operator would see a pre-warm took.Happy to write either up in more detail if it helps the unified design.