Repository navigation
Description
What happened?
create → resume → suspend → delete, every step successful, nothing force-deleted and nothing crashed — and the node still keeps /var/lib/ateom-gvisor/actors/<actor-UID>/ with the actor's durable-dir
data, checkpoints and bundle overlays in it.
Only atelet can delete that directory, and Terminate is the only RPC that asks it to. A plain delete never sends one.
Why
Two independent gates, either enough on its own.
- No worker assignment.
ensureAteletTerminatedreturns early whenstatus.worker_assignmentis unset:
substrate/cmd/ateapi/internal/controlapi/workflow_delete.go
Lines 121 to 125 in 27bf344
| assignment := actor.GetStatus().GetWorkerAssignment() | |
| if assignment == nil { | |
| slog.InfoContext(ctx, "actor has no worker assignment, skipping atlet terminate request", slog.Any("actor", actorRef)) | |
| return nil | |
| } |
That is the normal state, not an edge case. A plain delete only accepts SUSPENDED or CRASHED:
substrate/cmd/ateapi/internal/controlapi/workflow_delete.go
Lines 309 to 320 in 27bf344
| shouldDelete := false | |
| switch st { | |
| case ateapipb.ActorState_ACTOR_STATE_SUSPENDED, | |
| ateapipb.ActorState_ACTOR_STATE_CRASHED: | |
| shouldDelete = true | |
| default: | |
| // This allows deletion for any state | |
| shouldDelete = anyState | |
| } | |
| if !shouldDelete { | |
| return nil, status.Errorf(codes.FailedPrecondition, "Actor %s is not in a deletable state (state: %v)", actorRef, st) | |
| } |
and every transition into those states clears the assignment in the same commit —workflow_suspend.go#L412, crash.go#L101, workflow_pause.go#L282, workflow_revert.go#L251,
workflow_worker_delete.go#L217. Every actor that can be deleted has already lost the pointer the delete needs.
- Worker no longer hosts the actor. Even with the assignment retained, the delete still skips:
substrate/cmd/ateapi/internal/controlapi/workflow_delete.go
Lines 127 to 141 in 27bf344
| if workerName := assignment.GetWorker().GetName(); workerName != "" { | |
| // Ask whether the worker still HOSTS this actor, not whether its one | |
| // assignment happens to be this actor: a worker hosting several is the | |
| // ordinary case, and the others are none of this delete's business. | |
| hosted, err := workerHostsActor(ctx, w.store, workerName, actor.GetMetadata().GetUid()) | |
| if err != nil { | |
| return err | |
| } | |
| if !hosted { | |
| slog.InfoContext(ctx, "worker is no longer assigned to this actor, skipping atelet terminate request", | |
| slog.String("worker", workerName), | |
| slog.Any("actor", actorRef)) | |
| return nil | |
| } | |
| } |
ensureSuspendedFinalized calls releaseWorker before committing SUSPENDED, which drops the worker↔actor assignment row.
After a suspend commits, ateapi holds no pointer to the node at all: status.local_snapshot_info.node_vms_with_local_snapshots is the only other record of which nodes hold an actor's files, and suspend clears it in the same commit (workflow_suspend.go#L413).
Expected Behavior
Deleting an Actor reclaims its state directory on the node that holds it — whatever state it was deleted from, and whether or not the control plane still has a pointer to that node.
Steps to Reproduce
Reproduction
Two tests, no fault injection, branched off the commit above:
https://github.com/chw120/substrate/tree/probe-delete-assignment
TestDeleteActor_AteletTerminateGatedOnWorkerAssignment holds everything but status.worker_assignment fixed, against a fake atelet that records Terminate:
--- PASS: no worker assignment: the node is never told (0 Terminate received)
--- PASS: worker assignment present: the node is told (1 Terminate received)
The second row is the control — the fake atelet is reachable and does get called when an assignment exists.
TestSuspendThenDeleteNeverReachesAtelet runs the real ensureSuspendedFinalized, then deletes from SUSPENDED:
after suspend: state = SUSPENDED, worker_assignment = nil
delete logs: "actor has no worker assignment, skipping atlet terminate request"
atelet received: 0 Terminate
Commenting out toUpdate.Status.WorkerAssignment = nil at workflow_suspend.go#L412 does not change the outcome — the second gate fires instead:
delete logs: "worker is no longer assigned to this actor, skipping atelet terminate request"
atelet received: 0 Terminate
Sandbox Runtime
gVisor (runsc)
Agent Substrate Version / Commit SHA
Kubernetes Version & Environment
No response
Host OS & Architecture
No response
Relevant Logs and Diagnostic Output
Additional Context
No response
Confirmation
- I have searched existing issues and verified that this is not a duplicate.
- I have verified that this issue occurs on the latest commit on
main.
Activity
chw120 commentedon Sep 19, 2026
cc: Zoe Zhao (@zoez7)
aditya-shantanu commentedon Oct 7, 2026
Field data from a 3 TB
c3-standard-192-metalnode: 9,082 orphaned actor dirs / 894 GiB after one PauseActor ladder, and DiskPressure + atelet eviction within one morning (18,404 dirs / 2.79 TB) with 1.5 GiB actors. Details and code citations at 584d031 in #641 (comment).igooch commentedon Oct 21, 2026
🤖 Closing with #641. Both gates are gone on main. Delete no longer needs a worker assignment, and it sends a workerless
Terminatewhen the worker no longer hosts the actor (#2797). The remaining gaps are in #3098.Chenyi Wang (@chw120), please reopen if you still see a deleted actor's directory left on a node in a case #3098 doesn't cover.