Repository navigation
DeleteActor never cleans the actor's node directory #641
Description
Expected Behavior
Deleting an actor releases everything the actor held, including its node-local state: after
DeleteActor, /var/lib/ateom-gvisor/actors/<uid>/ should be removed (or at minimum contain
nothing that marks the actor as still placed on the node).
This matters doubly once #463 Phase 2 lands: the image-cache GC's root set is the per-bundle
rootfs-overlay.json specs — "spec on disk" means "actor placed here; do not evict its image."
A deleted actor should not root an image.
Actual Behavior
DeleteActor is a pure control-plane operation: it deletes the store record and cleans up volumes
(cmd/ateapi/internal/controlapi/delete_actor.go), but nothing ever tells the node. atelet's RPC
surface is only Run/Checkpoint/Restore, and resetActorDirs runs lazily — at the start of the
next Run/Restore or the end of a Checkpoint for the same actorUID. A deleted actor's UID
never comes back, so /var/lib/ateom-gvisor/actors/<uid>/ persists until the node is replaced.
What leaks depends on the state the actor was deleted from (the store allows SUSPENDED or CRASHED,
cmd/ateapi/internal/store/ateredis/ateredis.go):
- Deleted from CRASHED: a crash means no
Checkpointran after the lastRun/Restore, so the
entire bundle dir survives — the actor's writableupper/layer and the
bundles/<container>/rootfs-overlay.jsonspec. Post-Design: on-disk, layer-deduplicated OCI image cache for atelet #463-Phase-2, that leaked spec is
indistinguishable from a live one, so the dead actor's image is rooted forever: GC logs it as
in-use on every pass and it can never be evicted, regardless of disk pressure. Hit twice during
Phase 2 GC validation. The GC is behaving as designed (ambiguity must resolve toward retention —
evicting a mounted lowerdir is catastrophic); the defect is that deletion leaves the "I am placed
here" marker behind. Filed separately from Design: on-disk, layer-deduplicated OCI image cache for atelet #463 so this over-retention isn't mistaken for a GC bug. - Deleted after a clean suspend (the common path): the bundle was wiped at
Checkpoint, but the
ActorPathskeleton,sandbox-assets.json, and anylocal-checkpoint/contents are left
behind.
Steps to Reproduce the Problem
- On a Kind cluster with substrate installed, run an actor, then crash it (e.g. kill its
workload) so it lands in CRASHED. kubectl ate delete actor <name>— succeeds (CRASHED satisfies the delete precondition).- On the node:
docker exec <kind-node> ls /var/lib/ateom-gvisor/actors/<uid>/bundles/<container>/
→rootfs-overlay.json,upper/,work/,rootfs/still present, forever. - (Variant for the suspended path) Suspend an actor, delete it, and observe
local-checkpoint/
andsandbox-assets.jsonsurvive under the actor's dir. - (Post-Design: on-disk, layer-deduplicated OCI image cache for atelet #463-Phase-2) Put the node under disk pressure: GC logs the deleted actor's image as
rooted on every pass; it is never evicted.
Anything else we need to know?:
The fix belongs in the deletion path, not in GC: something should remove ActorPath(uid) when the
actor is deleted. Cleanup should key on deletion (the actor record no longer exists), not on
the CRASHED transition — whether a crashed actor should be recoverable from its node-local state
is an open design question on #292, and deletion-keyed cleanup is safe under either answer. Two
candidate approaches:
- Control plane calls atelet at delete time (a new teardown RPC). Direct, but the control plane
must still know — and be able to reach — the actor's last node when the delete happens. - atelet reconciles
ActorsDiragainst the control plane (periodic sweep removing dirs for
actors that no longer exist). Self-healing — it also catches missed deletes and actors crashed on
dead workers ([P3] releaseActorOnDeadWorker does not clear AteomPodUid — diverges from crashActor cleanup #618) — but it gives atelet a control-plane client it deliberately doesn't have
today.
A full "delete all" of the directory is safe: nothing can reference a deleted actor's node-local
state, because the actor record that would point to it is gone.
Related: #615 (node-side debris on an abnormal path), #172 (local snapshot storage management
umbrella).
Specifications
- Version: main @ e1fa004 (2026-07-28); GC interaction observed with the Design: on-disk, layer-deduplicated OCI image cache for atelet #463 Phase 2 POC branch
(phase2-gc-record-first@ a06ee86) - Platform: Kind (v1.36.1, Debian GNU/Linux 13 (trixie), 6.8.0-117-generic (arm64),
containerd://2.3.1) and GKE (v1.35.5-gke.1163012, c3-standard-4, 6.12.85+, containerd://2.1.7) —
behavior is identical on both
Activity
igooch commentedon Aug 12, 2026
Hold on this until #788 is merged.
igooch commentedon Sep 4, 2026
#788 (merged 2026-08-20) fixed the force-delete path; scoping the rest.
#788 implemented approach 1 (a teardown RPC):
atelet.Terminate→ateom.TerminateWorkload→resetActorDirs, wired into the delete workflow.resetActorDirsdoesRemoveAllWritableon the bundle dir, removingrootfs-overlay.json— so the GC over-retention concern is closed when the RPC fires.But it only fires for a live, reachable worker.
ensureAteletTerminatedskips whenWorkerAssignment == nilor the worker pod is gone — and both states a normal delete can touch clear the assignment first (suspend inFinalizeSuspended, crash incrash.go). So the two paths originally reported here still leak: delete-from-CRASHED and delete-after-SUSPEND. The dead-worker variant (#618) skips too.Remaining scope: clean node state when there is no live worker to call — approach 2 (atelet sweep of
ActorsDirvs. the control plane, or control plane targeting the last-known node).Note: even when terminate runs,
resetActorDirsrecreates subdirs (MkdirAll) and re-writessandbox-assets.jsonrather than deletingActorPath(uid). Fine if the bar is "nothing roots an image"; a separate sub-item if the bar is literally "ActorPathgone."Related: #618, #664.
igooch commentedon Sep 4, 2026
Small correction to the "open question" in the description: approach 2 (an atelet
ActorsDirreconcile) no longer requires giving atelet a control-plane client it lacks — on main atelet already dials ateapi and holdsActorIdentityClient+WorkerServiceClient(credential broker / worker-capacity). The list RPCs a sweep needs already exist too:ListWorkerActorAssignments(actors assigned to this worker) andListActors/GetActor(existence). So the remaining work is just the periodic loop: enumerateActorsDir, ask ateapi which UIDs still exist / are still assigned here,resetActorDirs(or removeActorPath(uid)) for the rest. The stated cost of approach 2 is stale.feat(api): introduce RevertActor lifecycle RPC (agent-substrate#1675)
zlammerts-svg commentedon Sep 23, 2026
Hi Ivy, are you still working on this one? I can see that both #618 and #664 are closed now? What work is left to close this issue?
igooch commentedon Sep 24, 2026
Zheng Lammerts (@zlammerts-svg) thanks for the follow up. Left comment in #1706 if they want to include the remaining scope of this issue in that PR.
aditya-shantanu commentedon Oct 7, 2026
Adding a field data point from scale testing, because this goes beyond slow inode growth: it takes a 3 TB boot disk to DiskPressure in one morning once actors carry real memory.
Environment: GKE,
c3-standard-192-metal, COS, 3 TB Hyperdisk boot disk. Substrateperf/resume-latency@584d0318; the control-plane and atelet code paths involved are unchanged from main (see citations).Observations
/var/lib/ate/actors/<uid>dirs, 894 GiB.ate-systemon that node; the node was unusable until the directories were removed by hand from a privileged node-critical debug pod.Code path confirming the mechanism (at 584d031, same as main)
Terminate->pruneLocalCheckpoints->removeActorDirs(cmd/atelet/main.go:1446-1453,removeActorDirsat:2016-2031, added in Remove an actor's directory when atelet terminates it #1654).ensureAteletTerminatedreturns early whenstatus.worker_assignmentis nil (cmd/ateapi/internal/controlapi/workflow_delete.go:128-132) or when the worker no longer hosts the actor (:134-148).workflow_pause.go:236releases the worker,:289nils the assignment), suspend (workflow_suspend.go:373releaseWorker,:416nils the assignment) and crash (crash.go:110-139) all clear the assignment in the commit that enters PAUSED/SUSPENDED/CRASHED, so every deletable actor is deleted without any component touching its directory.actors/; the only periodic reader is the image-cache GC (cmd/atelet/imagegc.go:82), which treats present dirs as pinning layers, so the leak also blocks layer eviction.docs/architecture.md:492("After deletion, the state of the actor (i.e., memory+disk snapshots) is garbage collected") holds only for the GCS prefix.docs/dev/scale-test-readiness.md:71,346,455tracks this as [Bug]: ateapi: a plain actor delete never tells the node, so the actor's state directory is never reclaimed #1757/Improve ate-apiserver / atelet contract #1778.Branch-only aggravation (not on main, FYI for anyone measuring on
perf/resume-latency): commit0ccab235adds--retain-uploaded-snapshots(defaulttrue,cmd/atelet/main.go:108; help text says "removed on terminate") which moves the uncompressed snapshot toretained-snapshot/(cmd/atelet/internal/ateletpath/ateletpath.go:74-77) after a suspend upload. Since Terminate is never reached on the delete-from-SUSPENDED path, suspend-then-delete also leaves the full memory image on the node. Mitigation there:--retain-uploaded-snapshots=false.Impact: any pause-path or long-running deployment leaks the full per-actor memory image per deleted actor until the node dies, and the leaked dirs pin image-cache layers. With 1 GiB+ actors this is hours, not weeks.
Proposed fix (consistent with the remaining-scope comment above and PR #1706):
status.local_snapshot_info.node_vms_with_local_snapshots(paused) and to the last-known node (suspended/crashed), best-effort so an unreachable node cannot wedge the delete./var/lib/ate/actors/<uid>for UIDs the control plane no longer knows (atelet already holds ateapi clients;ListWorkerActorAssignments/GetActorsuffice).TODO(dberkov)atcmd/atelet/main.go:1244).Short-term mitigations we are using: delete actors from RUNNING (assignment present, so Terminate fires), and on the branch
--retain-uploaded-snapshots=false.Found while running the agents-tco host-fill measurements (repo
adityashantanu/substrate-agents-tco). Happy to attach the nodedmesg/df/ducaptures if useful. Cross-ref: #1757, #1688, PR #1706.zlammerts-svg commentedon Oct 16, 2026
igooch Based on the latest comment, are we waiting for #1706 to be merged to close this one? Thanks!
igooch commentedon Oct 16, 2026
#1706 was partially blocked based on changes to the v2 lifecycle. We asked that the 1706 PR will be broken up into a couple of PRs, and one of the future PRs, not yet drafted, will resolve this issue.
igooch commentedon Oct 21, 2026
🤖 Closing, as main now reclaims an actor's node state in these cases.
Terminatewhenever the actor has anassigned_node, even when its worker no longer hosts it (ateapi,atelet: reclaim actor node state when a worker's pod is gone #2797).Terminate(ateapi,atelet: reclaim actor node state when a worker's pod is gone #2797).Two smaller gaps remain, tracked in #3098.