Repository navigation
Feature request: Fallback to local snapshot when failing to upload external snapshot. #362
Description
Current behaviour: when an Actor has configured SnapshotType_SNAPSHOT_TYPE_EXTERNAL but substrate fails to upload the external snapshot after checkpointg, we have to set the Actor to CRASHED state, because the most recent snapshot is lost.
Feature request: if we can fallback to storing the snapshot locally and upload later, we won't have to set the Actor to CRASHED state right away.
Activity
graceful termination for atelet (#748)
eren23 commentedon Oct 7, 2026
This has a real trigger. AX cancels
SuspendActorafter 15 s, theax suspendCLI deadline. With a workspace of about 1.5 GiB or more, the cancel lands during the durable dir upload, after ateom has already removed the sandbox. The retried suspend then fails onrunsc pauseand crashes the actor, and AX reverts it on the next resume, so the workspace is lost. I reproduced it on main0b91488d. Details in google/ax#453.eren23 commentedon Oct 7, 2026
Would you take a PR for the retry part of this? From what I read on main, most of it is already in place. After a failed upload, Checkpoint returns before
resetActorDirs, so the checkpoint dir with the durable dir tars stays on the node until the next Run, Restore or terminate. A retried Checkpoint then calls CheckpointWorkload again and fails because the sandbox is already gone.My idea is to write the snapshot file list to the sandbox record before the upload starts. A retried Checkpoint for the same actor would find it, skip CheckpointWorkload and only upload again. The actor stays SUSPENDING until that works, and terminate still cleans up as it does now. Resume does not change, so this is smaller than the local fallback in the title.
This overlaps with #372, and I saw troychiu offered to help there. If that work already covers Checkpoint, I will leave it.
eren23 commentedon Oct 9, 2026
I retested this on main 288694e. When a running actor suspends, a storage outage during the upload still crashes it and loses everything since the last snapshot, 2 of 2. Killing atelet or the new snapshot plugin sidecar during that upload does the same.
Pausing first and then suspending goes through UploadPausedCheckpoint, which keeps the local snapshot until the upload lands. When I killed the plugin 1 s into that upload, the call failed with Unavailable, the actor stayed SUSPENDING, and a second suspend finished with nothing lost, 2 of 2. A 60 s storage outage at the same point failed with Internal instead (
while putting object ... connect: connection refused), and handleAteletError crashed the actor. The local snapshot was still on the node, but from CRASHED the only way back is RevertActor, which drops it. So the new files were gone in 2 of 2.I first tried returning Unavailable for network errors in the plugin's toStatus, but that also catches TLS, DNS and credential failures and leaves actors stuck in SUSPENDING, which RevertActor does not accept, so I dropped it.
What looks cleaner to me is to put the actor back to PAUSED when UploadPausedCheckpoint fails and the local snapshot is still there, and to keep crashing when it is gone. As far as I can see the paused path only sets the state and InProgressSnapshotUri when it marks the actor SUSPENDING, so going back is small. Nothing is lost, nothing gets stuck, and the caller can suspend again later or resume. Would you take a PR along those lines, or do you have a different shape in mind?
Return a paused actor to PAUSED when its snapshot upload fails
eren23 commentedon Oct 10, 2026
I wrote the PAUSED idea up as a branch so it is easier to judge. It is eren23/substrate@main...eren23:substrate:fix/paused-upload-back-to-paused (one commit on main f200630).
When UploadPausedCheckpoint fails with anything other than NotFound, the suspend now puts the actor back to PAUSED with the same local snapshot and deletes the partial upload best effort. The actor can then resume or suspend again. NotFound still crashes it, and atelet now returns NotFound when the local snapshot is gone and no uploaded copy exists. Its interceptor used to send that as Internal. If the local snapshot is gone and the probe for an uploaded copy fails, atelet returns Unavailable, so the actor stays SUSPENDING and a retry can still find an upload that an earlier attempt committed.
In my kind lab I paused an actor, blocked TCP to the object store, suspended, unblocked and suspended again. On main f200630 the data written after the last durable snapshot was gone in 2 of 2 trials. With the branch the first suspend failed with "the actor is PAUSED again", the retry succeeded, and the data was intact in 2 of 2.
This flips TestSuspendActor_FromPaused_UploadFailureCrashes, which says the crash is intended, so I wanted to ask before I open a PR. I can open one if this direction works for you.
Pause a running actor before its suspend uploads the snapshot
aditya-shantanu commentedon Oct 20, 2026
Data point from a pre-GA chaos run on
main4b4fbb2 showing how cheap it is to hit this. Stopping theobjectstore-pluginsidecar of one atelet pod once (crictl stop, kubelet restarted it immediately, restartCount 0 -> 1) while 12 Locust users were doing sync suspends of 256 MiB gVisor actors crashed the two actors whose uploads were in flight on that node:Two things compound here. The checkpoint itself had completed (the sandbox was already gone), so the only thing lost was a retriable upload, which is this issue. And the plugin returned
Unavailable, but atelet wraps it withfmt.Errorfso the RPC reaches ate-api-server asInternal, whichateletErrorCrashesActortreats as fatal. Even without a local fallback, preserving the plugin's status code would have made this a retriable suspend failure instead of a crash. A sidecar restart is a routine event (OOM, image rollout, node pressure), so this is worth treating as a reliability bug rather than only a feature request.