Repository navigation
Recovery from failed resume, crash scenarios #643
Description
Related issue: #641
Once an Actor is in RESUMING status, the only allowed transition paths are:
- Success scenario: Actor becomes RUNNING
- Crash scenario: Actor becomes CRASHED
That means if some error happened during Resume, the only way out is for users to keep on retrying until it succeeds, or abandon and leak the actor.
#119 proposes a "revert" API, which could help with restoring to the last known state, but even that could also run into errors.
I think as a final escape hatch, we should provide some "force delete" option. where we can 1) trigger (perhaps periodically) a cleanup sequence on nodes and CP 2) still keep track of that actor and potentially leaked resources, but allow for new actors with the same name to be created.
To accomplish this, we probably have to:
- Track force deleting actors under a different key, maybe we prefix the actor UID with some keyword like "forcedeleting"
- Have periodic garbage collectors in both CP and atelet.
- Atelet only GCs actor UIDs that have been marked as forcedeleting.
- CP GC waits for atelet GC for that actor UID to complete. This implies that we need to keep track of which atelet the actor was running on and also that we wouldn't have allowed the actor to leak on multiple nodes.
Any other thoughts/ideas?
Activity
Make UnmountVolume a hard failure during suspend/pause.
Make UnmountVolume a hard crash failure during suspend/pause.
Make UnmountVolume a hard crash failure during suspend/pause. (#666)
msau42 commentedon Aug 3, 2026
Untested POC here: main...msau42:substrate:force-delete
For now I decided to focus on adding just a cleanup workflow, but not solve for the "allow creating same actor name while deletion is in progress" problem.
zoez7 commentedon Aug 11, 2026
yeah this is going to be challenging, for now maybe we can just return an error with "previous actor is still in wipeout, cannot create actor with same name".
Sneha-at commentedon Aug 14, 2026
There are 2 issues here
#788 tracks the 1st point in this issue.
Allow deleting an actor from any state. (agent-substrate#788)