A crash leaves the actor's sandbox running on its worker; a gVisor re-host then loses track of it #2355

Context

When a lifecycle call fails, ateapi can crash the actor (crashActor, cmd/ateapi/internal/controlapi/crash.go). A crash only updates ateapi's store: it releases the actor from its worker, marks it CRASHED, clears its worker assignment, and emits ate.actor.crashed. It makes no atelet call, so nothing tells the worker to stop the actor.

Issue

1. A crash can leave the sandbox running on the worker. A checkpoint that fails before its teardown returns before terminateWorkload. On gVisor that includes the pause, runsc checkpoint, and durable-dir tar failures; the micro-VM has the pause and snapshot failures. The sandbox keeps running and the ateom keeps the actor hosted. ateapi then crashes the actor:

• The worker's slot is free in ateapi's store, but the ateom still counts the actor against its own --max-actors.
• Deleting or reverting the crashed actor does not stop it: with no worker assignment, ensureAteletTerminated sends atelet no target ateom, and atelet skips TerminateWorkload.
• The sandbox runs until the actor is re-hosted on that same worker pod, or the pod goes away. This holds for both runtimes.

2. A re-host handles the leftover differently per runtime. Run and Restore can find the actor still hosted: after case 1, if ateapi places the actor on the same worker pod, or on a retried call whose first attempt succeeded.

On gVisor the old sandbox then has no session, no record, and no way for runsc to reach it. The cgroup leaf is named only by actor UID and container name; if the new sandbox shares the old one's leaf, the new activation, which counts from zero, also counts the old sandbox's CPU in the usage records and atelet's CPU counter.

Proposal

1. Stop the workload when crashing an actor that may still run. When the crash follows an atelet error, crash after asking atelet to terminate the workload on the assigned ateom, or keep the target so a later Delete or Revert can. This needs a decision on retries, since ateletErrorCrashesActor already skips crashing on Terminate errors.
2. Make gVisor match the micro-VM on a re-host: before the runsc state is reset, stop and delete the old sandbox's containers through its saved session, write the old activation's final usage record, and fail the RPC if the stop fails.

(2) fixes the re-host case on its own; (1) fixes the leak on pods the actor never returns to.

Related

The crash case shares a root cause with these: transitions into CRASHED (and SUSPENDED) clear WorkerAssignment, so later node-side cleanup has no worker to target.

• Ensure Atelet API is idempotent #372: idempotent atelet Run, Checkpoint, and Restore; covers the retried-call re-host.
• Actor crash leaves external volumes attached to the node; delete then skips detach #1715: a crash leaves external volumes attached; Delete then skips the detach.
• [Bug]: ateapi: a plain actor delete never tells the node, so the actor's state directory is never reclaimed #1757: a plain delete never tells the node, so actor directories leak.
• DeleteActor never cleans the actor's node directory #641: DeleteActor never cleans the node directory; a crashed actor leaves its whole bundle.
• atelet: actor state directories are never reclaimed when a worker pod goes away #1688: actor state directories are not reclaimed when a worker pod goes away.

docs/architecture.md and docs/api-guide.md say RevertActor terminates any bound worker sandbox. For a CRASHED actor the assignment is already cleared, so it cannot.

Found in review of #2352. cc Julian Gutierrez Oschmann (@juli4n), since #1715 notes the actor/worker detachment contract is being worked on under #372.