Repository navigation
OOMKill can take down entire worker #3092
Description
An OOM kill inside a worker pod can kill the whole worker and every actor on it. This happens two ways.
1) One actor going over its limit, on nodes with runc 1.5.2+ and systemd 253 through 258.
runc 1.5.2 sets OOMPolicy=kill on the container's systemd scope when kubelet asks for memory.oom.group=1, which kubelet does by default on cgroup v2 (you can disable this via a kubelet config option, but that's not very portable).
systemd 253-258 then stops the whole scope on any OOM kill under it, because it watches the scope's hierarchical oom_kill count. The kernel confined the kill to the actor's cgroup, systemd kills everything else.
systemd 259 fixed this by reading oom_group_kill from memory.events.local for these units:
Current distros are in the affected systemd range:
Details
| Distro | systemd | Leaf OOM kills worker with runc 1.5.2+? |
|---|---|---|
| Ubuntu 22.04 | 249 | no |
| Ubuntu 24.04 LTS | 255 | yes |
| Ubuntu 25.10 | 257 | yes |
| Ubuntu 26.04 | 259 | no |
| Debian 12 | 252 | no |
| Debian 13 | 257 | yes |
| RHEL 9 family (CentOS Stream 9, Alma 9, Rocky 9) | 252 | no |
| RHEL 10 family (CentOS Stream 10) | 257 | yes |
| Fedora 42 | 257 | yes |
| Fedora 43 | 258 | yes |
| Fedora 44 | 259 | no |
| openSUSE Leap 15.6 | 254 | yes |
| openSUSE Tumbleweed | 261 | no |
| Arch | 262 | no |
| Amazon Linux 2 | 219 | no |
| GKE COS | 254 | yes |
If you are using runc 1.5.2+ (current stable) and have systemd cgroup driver enabled (default in e.g. kubeadm) then you're impacted by this. That includes GKE rapid, and it will include kind once runc is updated.
This hits gVisor, where the actor's cgroup has memory.max at its memory limit. Micro-VM guests OOM inside the guest.
substrate/internal/sizing/sizing.go
Line 96 in 1ab66d6
2) The worker pod hitting its own memory limit, on any node.
kubelet's memory.oom.group=1 makes the kernel kill the whole container when the container itself runs out, so this doesn't depend on runc or systemd versions. The main way we hit it is suspend: snapshots are written through the page cache, charged to the worker pod with no limit of their own. On GKE, 2 filled 256Mi gVisor actors suspending together in a 512Mi pod OOM'd the worker, as did 4 in 1088Mi. Micro-VM snapshot writes OOM tight pods the same way.
substrate/cmd/ateworker-gvisor/ckptcgroup.go
Lines 38 to 39 in 1ab66d6
substrate/cmd/ateworker-microvm/checkpoint.go
Line 279 in 1ab66d6
The pod can't turn either off. OOMPolicy is fixed when runc creates the scope and is only reachable over the host's D-Bus, and node settings (kubelet singleProcessOOMKill, systemd 259+) aren't something we can require.
What I think we should do
- for 1), never let the kernel OOM-kill inside the worker: gVisor actor cgroups get
memory.highat the limit instead ofmemory.max, and the worker kills an actor that stays over it withcgroup.kill, which systemd doesn't count as an OOM. This could be dropped once nodes move to systemd 259+ ... but that might be a while. - for 2), cap snapshot page cache with
memory.highwhile it's written, unless the pod has enough unbooked memory free to lend it, and have workers keep some of their memory limit out of the capacity they report, see Fix Actor Memory Overhead Model #3091
I have prototypes of both, tested on GKE. We should ship this alongside #3091 for the better memory reservation API because ateworker will start OOMKilling based on it.
Activity
thockin commentedon Oct 21, 2026
There's a lot of directions we can take this, and the behaviors and interactions can be very subtle. I agree that this is a bad situation. Modelling overhead seems correct to me and unlikely to explode things.
These seem much more subtle. How bad would it be if we did this part in a 1.1 release O(weeks) later?
EItanya commentedon Oct 21, 2026
Agreed.
1 really scares me, and I don't have a great answer off the top of my head. Just to be clear this will affect gVisor and uVM? The distro versions seem right in the sweet spot of what folks are probably running.
2 follows a bit of a pattern, there were also recently oomkills on the snapshot transfer component. I'm worried generally also about reserving too much memory on top of what the actor is requesting, with such a high density system it adds up quickly. Just spitballin' here, can we potentially dodge memory altogether and write directly to a file, or create a shared pipeline of snapshot operations for all actors on that worker. Obviously those have their own fairly large downsides, but it feels to me like the answer here may be a set of options which the user can decide on based upon the style of actors they're running. What do you think?
BenTheElder commentedon Oct 22, 2026
Welll GKE 1.37 can pretty easily have the whole worker go down with no other fix in site. I already have implementations, just didn't want to start code dumping without discussion.
Right now this only impacts gVisor, uVM the user workload is inside the guest where THAT oomkill applies distinctly.
However this is because we don't constrain the cloud-hypervisor or virtiofsd ... the only reason those are impacted is we don't set a memory limit on those cgroups currently, which sills them to triggering case 2.
So ... no, but yes.
Yeah, basically if you're on a common distro AND you have runc up to date you are probably impacted. Plus at least GKE.
If you have very recent or very old systemd, don't use the systemd cgroup driver, or you have outdated runc you won't hit the first case ... but it's a relatively common combination that probably isn't going anywhere fast.
That's basically it: when the worker is tight on space, force it to reclaim page cache from the process, otherwise we can be more relaxed, which will be down to how much memory we reserve in #3091 we can tune it.
BenTheElder commentedon Oct 22, 2026
Here is what mitigating 1) looks like, it's really not bad: #3106
A gVisor actor going over its memory limit no longer takes down every…
Suspending actors no longer OOM-kills tight worker pods (agent-substr…
A microVM actor using more host memory than it booked no longer takes…
destijl commentedon Oct 29, 2026
Resolved on
mainby two fixes:f9aec710) placed each gVisor actor into a dedicated child cgroup withmemory.highthrottling andcgroup.killteardown onmemory.maxOOM events instead of relying on systemd scopeOOMPolicy=killacross the worker pod.e8cdf13d) throttled snapshot dirty page writeback duringSuspendActor(memory.high) and dropped page cache in 8 MiB chunks (POSIX_FADV_DONTNEED) so concurrent snapshot writes cannot exhaust worker pod memory and trigger a pod-level cgroup OOM kill.destijl commentedon Oct 29, 2026
Oh my bad Benjamin Elder (@BenTheElder). I was doing an issue cleanup and the agent thought this one was done.
destijl commentedon Oct 29, 2026
Reopening: keep open while remaining M3 memory reservation and worker OOM tracking is in flight.
BenTheElder commentedon Oct 31, 2026
Technically this is fixed but we are doing some hardening, this was ... very complicated to ship at the last moment and entirely too agentic as a result.