atelet: stale ateom directories are never garbage collected; every deleted worker pod leaks its socket dir onto the discovery scan #1677

What happened?

Nothing ever removes /var/lib/ateom-gvisor/ateoms/<worker-pod-UID>/ when a worker pod is deleted. Every ateom creates its directory (holding exactly one file, ateom.sock) at boot; the only cleanup anywhere in the tree is an ateom RemoveAll-ing its own stale socket when the same pod restarts. Directories accumulate until node replacement, and every pool rollout replaces all worker pods with fresh UIDs — so growth is rollout-driven and linear in node lifetime.

This was rational neglect, not a decision: repo archaeology (issues, PRs, commit history) found no discussion, because the directory set had no reader — atelet dialed sockets by exact UID from control-plane state, and a dead pod's leftover directory cost a few idle bytes. That changed when the stats sweep (#961) promoted the directory listing into its discovery registry. Since then every stale directory costs a dial+probe per sweep per node, debug-log noise, and (since #1479) a cachedPools entry. The only written acknowledgment is the comment that shipped with the sweep: "stale directories left by deleted worker pods (nothing garbage-collects them eagerly)". #1479's description references this issue as the durable fix.

Expected Behavior

Stale ateom directories are garbage collected. Two complementary mechanisms:

1. atelet janitor (the necessary mechanism — covers SIGKILL, OOM, node-crash leftovers). New cmd/atelet/ateomgc.go modeled on the imagegc.go loop/flags/seams pattern. The evidence rule must make false positives structurally impossible, because the blast radius of wrongly deleting a live ateom's directory is severe and self-hiding: the bound unix listener keeps serving established connections on the unlinked inode with no error, but every NEW connect fails ENOENT forever (nothing re-binds the socket), while the worker keeps advertising capacity — a semi-permanent worker blackhole with loss of checkpoint/stop control over its actor. Per pass, per directory entry:

1. Names must parse as UUIDs; skip and warn anything else (operator debris is not ours to delete).
2. The janitor takes its OWN node-scoped pod LIST (spec.nodeName field selector, NO label selector — a pod that lost its pool label must not become deletion-eligible). A list ERROR aborts the whole pass: no strikes advance, nothing is deleted. This is deliberately not the stats sweep's fetchWorkerPools, whose nil-means-failed collapse is right for label enrichment and disqualifying for deletion evidence.
3. UID present in the list -> reset its strike counter. A terminating pod stays listed until kubelet confirms container death, and a booting pod is in etcd before its container starts, so live/draining/booting ateoms are never candidates.
4. UID absent -> probe the socket (short-timeout GetActiveWorkloadStats, the sweep's probe shape). ANY successful response — even NO_SAMPLE_REASON_NO_WORKLOAD — vetoes and resets: a partial-but-successful list (a real failure mode, see atelet: cache pool resolutions so a list flap cannot split the CPU counter #1479's tests) cannot cause damage, because you cannot delete a directory whose ateom answers.
5. Otherwise strike++. RemoveAll only at N=3 consecutive passes AND directory mtime older than a min-age (~10m). Deleting a live directory would then require an ateom simultaneously absent from three independent quorum LISTs and unresponsive to three probes — dead by definition.
6. Flags per the imagegc precedent: --ateom-gc-period (default ~10m, 0 disables), min-age, --ateom-gc-dry-run.

Both existing directory readers already tolerate mid-scan ENOENT (the sweep's "not a target this tick" rule; the dialer is lazy), so deletion needs no coordination with them.

2. ateom self-cleanup on graceful shutdown (cheap complement, keeps steady state near zero). defer os.RemoveAll(ateomDir) after the successful MkdirAll in both runtime mains. Safe against same-UID restarts because kubelet serializes them (the replacement container does not start until the old process — and its defer — has exited). Rollouts, the dominant growth driver, terminate gracefully, so the janitor's caseload shrinks to ungraceful exits.

Steps to Reproduce

1. On any node, note the entries under /var/lib/ateom-gvisor/ateoms/.
2. Roll a WorkerPool (e.g. bump its ateomImage): every worker pod is replaced with fresh UIDs.
3. Observe the old pods' directories remain, each holding a dead ateom.sock; the stats sweep dials each one every interval and debug-logs the failure, forever.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

main

Additional Context

• Deleting stale directories is also what prunes the stats sweep's cachedPools (atelet: cache pool resolutions so a list flap cannot split the CPU counter #1479), whose entries are currently retained for as long as the directories are.
• Out of scope, worth a separate audit: /run/netns/ateom:<uid> accumulates by the same per-pod-UID pattern, outside BasePath.
• atelet state persistence: recover per-actor state across restarts #1372 (atelet state persistence) considered enumerating ateom directories for restart recovery; its accepted direction is ateom-initiated reconnection, but if that ever changes, the janitor's evidence rule (pod-gone AND socket-dead) already guarantees recovery only ever loses entries with no live ateom behind them.

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.