Repository navigation
Delete misses node state after every detach report fails until give-up #3098
Description
After #2797 and #2871, one case can still leave an actor's files on a node after DeleteActor.
#2871 keeps snapshot-state.json until atelet's detach report is acknowledged or refused with NotFound. If the control plane has recorded the detach, it has cleared assigned_node, so a later delete skips Terminate. Normally atelet's next report gets NotFound and removes the file. But if every report fails until atelet gives up, for example during a long outage between atelet and ate-api-server, the file and the actor's directory stay on the node.
Activity
cloudonly commentedon Oct 22, 2026
Tested at 3e2797bc, with production code unchanged. The results narrow both cases:
The normal sync-suspend path did not reproduce case 1. When atelet kept a local copy, both
assigned_nodeandlocalityremained set, andDeleteActorsent a workerlessTerminate. A real atelet test confirmed that this immediately removed the actor and snapshot directories. When no local copy was kept, both fields were empty. Directly clearingassigned_nodein persistence while keepinglocalitydid reproduce the skippedTerminate, but that was an injected state, not the result of the normal suspend. The existing sync-suspend regression also checks that both fields remain set.Failed detach reports did retain
snapshot-state.jsonand its directory. This reproduced for explicit and background detach withUnimplemented,FailedPrecondition,InvalidArgument, andUnavailableuntil give-up. The checkpoint directory and sandbox asset links were already removed. Restarting the tracker and advancing its fake clock by 24 hours scheduled no further report or detach. An explicit detach retry followed byNotFoundremoved the remaining state. This matches the existing detach-report tests.However, a failed report did not prevent later cleanup when the control plane still retained the node binding. The API test sent
Terminate, and separate real atelet tests ranCheckpoint -> Detach -> failed report -> workerless Terminatefor all four errors. Each removed all actor and snapshot directories. Retained state after detach therefore does not by itself demonstrate aDeleteActorcleanup failure.The remaining scenario to reproduce end-to-end is a detach committed by the control plane whose acknowledgements never reach atelet, followed by report give-up and deletion. I verified separately that a committed detach clears the node binding and makes Delete skip
Terminate, and that node-side give-up retains the file. I did not reproduce that combined network failure.Targeted tests used
-mod=readonly -race -count=3 -shuffle=on: 126 passing test/subtest events including repeats, no failures, skips, or race reports. Coverage included the linked regressions,TestDeleteActor_Paused,TestTrackerDetachCleansUpActorDirectory,TestImmediateCheckpointKeepsLocalCopy,TestTerminateWithoutWorkerPodUID, and additional local, uncommitted diagnostic tests. API tests used PostgreSQL/envtest with a fake atelet; node tests used production atelet/tracker code, real files, and fake workers/object storage. No cluster E2E was run.[-]Delete misses node state for a sync-suspended actor or a failed detach report[/-][+]Delete misses node state after every detach report fails until give-up[/+]igooch commentedon Oct 22, 2026
🤖 Thanks, that's right on case 1. A sync suspend keeps
assigned_nodewhenever atelet keeps a local copy (workflow_suspend.go:410), so delete reaches the node. I've removed it from the issue. Case 2 leaks only when atelet's reports fail until give-up after the control plane has recorded the detach. Otherwise the next report getsNotFoundand removes the state file. I've narrowed the issue to that case.