atelet: actor state directories are never reclaimed when a worker pod goes away #1688

What happened?

Nothing removes /var/lib/ateom-gvisor/actors/<actor-UID>/ when the worker pod hosting the actor goes away. The directory holds the actor's durable-dir contents, checkpoint/restore state and bundle overlay — for a durable-dir workload that is gigabytes per actor, not the few idle bytes of #1677's socket dirs. The directories persist for the life of the node.

This surfaced as node disk climbing during a durable-dir benchmark and never falling afterwards, eventually reaching ~90% and getting atelet evicted.

Measured on a 3-node dev cluster (c3-standard-4, 100 GB pd-balanced) against a0d306ee. Workload: 15 concurrent clients, each driving an actor that writes a 1 GiB file into its durable dir and then suspends and resumes repeatedly, for 10 minutes.

Reproduced on both sandbox classes; they share ateompath.BasePath, so the layout and the leak are identical. An equivalent micro-VM run left 18.33 GB across 16 dirs, likewise unchanged by scaling to 0.

No delete is required to trigger this — an actor being de-scheduled from a node is enough.

Mechanism

As far as I can tell, two independent gaps compose:

1. The worker-delete path never contacts the node. WorkerPoolSyncer.reconcile notices the pod is gone and calls Control API DeleteWorker. WorkerWorkflow.ensureBoundActorsReleased → releaseBoundActor moves each bound actor to CRASHED and clears its pod pointers, but nothing asks atelet to terminate it. After the worker record is deleted, the actor names no worker, no worker names a node, and nothing ever revisits the actor UID — the directory is orphaned permanently.

2. The actor-delete path gives up when the pod is gone. ActorWorkflow.ensureAteletTerminated dials DialForWorker(ns, pod) and returns nil on ErrWorkerPodNotFound. That is the correct shortcut for a pod that legitimately vanished, but it is also exactly how the state comes to be orphaned.

Even when a Terminate is issued, it fails before reaching the directory cleanup: the teardown dials the ateom socket first, and a vanished ateom (connect: no such file or directory, connection refused) aborts the call ahead of the unmounts and the directory removal.

Expected Behavior

Actor node state is reclaimed once no actor can use it again. Two complementary mechanisms, mirroring the split proposed in #1677:

1. Prompt reclaim on the delete path. The worker-delete workflow is the last moment anything still knows which node holds the state, so it should tell that node to reclaim before releasing the record. Reaching the atelet requires dialing by node rather than by pod, since the pod is precisely what has disappeared. This must be best-effort: an unreachable node cannot be allowed to wedge worker deregistration.

   Terminate also needs to tolerate a vanished ateom — skip the sandbox teardown and still run the unmounts and directory cleanup — otherwise the call fails before doing the part that matters. A reachable ateom that rejects the call should still fail: that sandbox is live.

2. A sweep as backstop. Evictions, node crashes and uninstalls produce orphans that no delete path will ever visit. A periodic atelet sweep reconciling actors/ against the actors the control plane places on that node covers them. It must fail closed — if the live set cannot be read in full, delete nothing — and must skip actors holding a local pause snapshot, which is node-pinned state whose deletion is unrecoverable.

Neither covers the graceful path. Reclaiming during Checkpoint as well would keep steady state near zero and shrink the sweep's caseload; that is a separate change.

Steps to Reproduce

1. Install ate-system and deploy a WorkerPool with 15 workers.
2. Record the node state:

   du -sb /var/lib/ateom-gvisor/* && ls -1 /var/lib/ateom-gvisor/actors | wc -l

3. Drive 15 concurrent actors for ~10 minutes, each writing a 1 GiB file into its durable dir and suspending/resuming between iterations.
4. Record the node state again — expect several GB per node under actors/.
5. Scale the WorkerPool to 0, wait for the pods to drain, and record again.

Every actor directory and every byte is still present. The same holds after hack/install-ate.sh --delete-all and after a fresh reinstall.

Sandbox Runtime

Both / Runtime Agnostic — gVisor and micro-VM share ateompath.BasePath.

Agent Substrate Version / Commit SHA

main @ a0d306ee

Additional Context

Not fixed by #1678. That PR adds defer os.RemoveAll(ateomDir) to the two ateom runtime mains, which covers ateoms/<pod-UID>/ only. actors/<uid>/ is written and owned by atelet, not by ateom, and is not touched by that defer — nor could it be, since the ungraceful exits that strand actor state are exactly the cases where a defer does not run. Same pattern, different directory, very different blast radius:

#1677's proposed janitor (cmd/atelet/ateomgc.go) is close in shape to the sweep described above — both modeled on imagegc.go, both node-scoped, fail-closed, with a min-age. If both land it is worth deciding whether they should be one loop.

Documentation gap. There is no written description of atelet's on-disk layout or how it does bookkeeping, which is part of why the ownership and lifetime of these directories is unclear.

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.