[bug-bash] microVM: OOM kill inside the guest is silent, actor stays RUNNING #3615

On microVM, when a container goes over its memory limit (or the guest runs out of RAM), the guest kernel kills it. Substrate doesn't notice. No log, no crash, the actor stays RUNNING. The app output just stops.

A customer benchmark hit an OOM and didn't know what happened. On gVisor this case is now reported (#3106): the actor goes to CRASHED with "killed for exceeding its memory limit". microVM has no equivalent for OOMs inside the guest. #3429 only covers the VMM using more host memory than it booked.

Repro

Tested on GKE at main fc931c4. Template with two containers, 512Mi actor limit:

• memhog (has the wakeup probe), no container limit
• capped, with a 128Mi container limit

Then:

1. Make capped allocate past 128Mi: the guest kernel kills it.
2. Make memhog allocate past guest RAM: the guest kernel kills it.

Both times:

• the actor stays RUNNING, even after memhog (the container with the wakeup probe) is dead
• no worker log, no crash report, no Actor crashed event
• kubectl ate logs actor just stops
• the only sign later is an atunnel CONNECT upstream failed WARN when a request comes in

Why

• We never call kata's GetOOMEvent, so guest OOM kills are not seen.
• We don't watch container processes while the actor runs. WaitProcess is only used in graceful shutdown (cmd/ateworker-microvm/shutdown.go).
• The guest kernel log (where the OOM killer writes) goes to console.log only. We read it only when the agent fails at boot, and cleanup deletes it at teardown.

Proposal

1. Poll GetOOMEvent for each running actor. On an event, log a WARN with ateattr.ActorLogAttrs and the container name.
2. Decide what the actor state should be when a container is OOM-killed. Probably the same as [Bug]: An actor stays RUNNING after its only container exits #2972 (container exits → CRASHED), with "killed for exceeding its memory limit" as the message so it matches gVisor.
3. Log the tail of console.log with actor labels when the VM dies or a container is killed, before cleanup deletes it.

Related

• [Bug]: An actor stays RUNNING after its only container exits #2972: actor stays RUNNING after its only container exits (state side of this)
• Add an ate.actor.oom_kills metric for gVisor and microVM #2427: ate.actor.oom_kills metric
• Make sandbox crashes visible in logs and metrics #2212 item 6: log console.log tail before cleanup
• ateom sandbox runtime output (runsc, pause container, cloud-hypervisor) has no actor attribution #2271: cloud-hypervisor output has no actor labels
• [bug-bash] Track gVisor / microVM parity gaps for 1.0 #3226: gVisor / microVM parity gaps for 1.0