Repository navigation
Reanme ateomImage as workerImage for WorkerPool #861
Description
If customer has their own ateom implementation, they can provide one in ateomImage.
Otherwise this field should be optional, and leaving it empty will let substrate system (specially atecontroller) inject a versioned ateomImage, based on the sandboxClass to pick ateom-gvisor/ateom-microVM image that atecontroller carries.
This is a critical part of substrate system upgrade, it doesn't make senses to edit customer's CR for every ateom upgrade.
Activity
HavenXia commentedon Aug 11, 2026
Benjamin Elder (@BenTheElder) Let me know what's your opinions.
HavenXia commentedon Aug 24, 2026
Status update
Discussed this with Benjamin Elder (@BenTheElder) in the context of the substrate upgrade proposal, Where it stands now:
The original shape of this issue still holds, when the image field is not set, the system injects a versioned default based on
sandboxClass.Open concerns:
atecontrollerbinary or through a flag, patching an worker image CVE requires rollingatecontroller. Dataplane seems patch happen much more often than control plane changes, so per-component upgrade must stay possible.Direction we are converging on, but need more discussion:
1. My idea: Make dataplane rolling aate-controller-owned behavior. On any change to a WorkerPool CR,ate-controllerkeeps two worker sets, drains nodes one by one, and fills in new workers. A system upgrade driver triggers it the same way a customer would.2. To address concern 1, the injected default may not live in the controller. Instead it can live in a versioned config object that
atecontrollerconsumes (a ConfigMap or a small release/sandbox-class CR). A worker Image CVE patch is then just a write to that object: the controller binary see it and performs the safe rolling. The part of contorller logic is shared with the full system upgrade.cc Tim Hockin (@thockin) for the format
HavenXia commentedon Aug 24, 2026
In fact, the idea of atecontroller driving the node draining one by one might also serve as a core functionality solution for smooth sandbox runtime upgrade? cc Zoe Zhao (@zoez7)
sfunkenhauser commentedon Aug 24, 2026
No, I don't think anybody has thought about this yet, but this likely already works with the default deployment rollingupdate strategy. Some new workers are required to be available before starting to evict the old ones, and a max number of pods can be unavailable at a time. For a saturated workerpool we might need to tweak these values (reduce maxUnavailable) to ensure there is a new worker available for each actor.
There is also the question of how does an actor resume after it's worker / pod is evicted. Currently this needs to be handled by the harness, but I assume eventually we might want to consider 'live actor migration'.
HavenXia commentedon Aug 25, 2026
I think eviction with the harness handling recovery sounds acceptable for the involuntary cases.
My point is the voluntary one: a user legally edits their own WorkerPool (image, pod shape or anything else), should we rely on the
default rolling update to proactively suspend all actors(
SIGTERMpropagated into actors, harness suspends actors with communication), orallow a passively upgrade that users can suspend on their own pace (like substrate upgrade proposal)
My understand this choice should be the same one we make for the dataplane rolling in the system upgrade, whichever way it goes. Benjamin Elder (@BenTheElder) has been advocating the passive one there.
sfunkenhauser commentedon Aug 25, 2026
Maybe I misunderstood your question, but I interpreted it to be referencing worker pool availability (old workers stay intact until new workers are available). There are built in K8s mechanisms to ensure availability during an upgrade / update.
This is another matter, and I agree that creating a new deployment and draining the old one is likely the easiest approach for the short term.
That being said pod evictions aren't completely avoidable, and actors / their harness should be able to handle them gracefully. If this is the expectation, why not rely on pod evictions during worker pool updates?
HavenXia commentedon Aug 25, 2026
The disucssion has been diverged to another topic about UX of workerpool upgrade & system upgrade.
I think for the original question we already have an answer that we will modify the field as an optional
workerImage.workerImageover ours, they can put it inworkerImageand they are responsible for its compatibility with other components.workerImagewill be provided by the install/upgrade script and injected into workers byate-controller.api: rename ateomImage to workerImage and make it optional (#1210)
[-]Make `ateomImage` an optional field for `WorkerPool`[/-][+]Reanme `ateomImage` as `workerImage` for `WorkerPool`[/+]HavenXia commentedon Aug 31, 2026
After offline discussion, we should not make workerImage optional as it provide explicit information for what image workers use, removing it will need
atecontrollerto inject workerImage and leads to unnecessary control plane & dataplane wiring.api: make WorkerPool workerImage required again (agent-substrate#1334)