Support suspending a PAUSED actor without waking it

From #119,

If an actor remains paused for an extended period and --timer is provided, the system escalates it to a SUSPENDED state to ensure data safety and free up local node storage.

We need to support the SuspendActor on a PAUSED actor, overall it should be one atelet call that uploads the existing local files, then the api records the ActorSnapshot, clears LocalSnapshotInfo and the node pin, and sets SUSPENDED.

This is a critical path for both Actor Lifecycle(#119) and System Upgrade(#473)

Proposed implementation plan, split into 5 PRs so work can proceed in parallel (PRs 1–3 are mutually independent; only PR 5 turns the feature on).

Design summary

Keep the existing 5-step SuspendActor workflow and branch inside the steps on a FromPaused discriminator computed at load time: status == PAUSED, or status == SUSPENDING && WorkerAssignment == nil && LocalSnapshotInfo != nil (the nil-assignment conjunct is required because resume never clears LocalSnapshotInfo, so a RUNNING actor resumed from pause still carries a stale one).

On the node, a new atelet RPC UploadPausedCheckpoint (not a new CheckpointType — the Checkpoint handler drives a live ateom, which a paused actor doesn't have) reads the local pause snapshot's manifest.json, uploads the files via the same ategcs primitives as uploadExternalCheckpoint (manifest last, as commit marker), then prunes the local checkpoint. ateapi dials atelet by node name (LocalSnapshotInfo.NodeVmsWithLocalSnapshots) via a new DialForNode, since a paused actor has no worker pod.

Scope handling — the upload honors the commit scope (OnCommit), like a normal suspend:

Captured (OnPause)	Desired (OnCommit)	Behavior
FULL	FULL	plain upload
DATA	DATA	plain upload
FULL	DATA	microVM: extract durable-disk files; gVisor: `Unimplemented` until split-checkpoint runsc (#790)
DATA	FULL	rejected (`FailedPrecondition`) — cannot fabricate memory; covers the golden-atespace corner

The captured scope is recorded in the snapshot's own manifest.json (new scope field written at Checkpoint — authoritative at upload time) plus a control-plane copy in LocalSnapshotInfo.content_scope for early rejection before the actor leaves PAUSED.

Failure semantics: dial/upload failures are retryable (actor stays SUSPENDING, client-driven forward recovery; local files intact, deterministic object names). Only definitive data loss (local dir gone AND no remote manifest) crashes the actor (LOCAL_SNAPSHOT_GONE). Idempotent retry after a successful upload is detected via the remote manifest probe.

PR breakdown

PR 1 ──(soft)──> PR 4 ──┐
PR 2 ───────────────────┼──> PR 5
PR 3 ───────────────────┘

PR 1 — Record scope in the snapshot manifest (atelet only)
Add Scope to sandboxAssetsRecord; Checkpoint writes it for local and external manifests. Tiny, no API change, no dependencies.

PR 2 — LocalSnapshotInfo.content_scope (control plane only)
ateapipb proto + regen; FinalizePausedStep records the OnPause scope. No dependencies.

PR 3 — Behavior-preserving refactors (control plane only)
Dialer: extract dialAteletOnNode, add DialForNode + ErrNoAteletOnNode. FinalizeSuspendedStep: hoist finalization (ActorSnapshot creation + SUSPENDED flip) out of the if assignment != nil block — latent bug: with a nil assignment the workflow would "succeed" leaving the actor SUSPENDING forever. No dependencies.

PR 4 — atelet UploadPausedCheckpoint RPC (dead code until PR 5)
ateletpb proto (snapshot_prefix, snapshot_uri_prefix, desired_scope, spec) + regen; handler with manifest read, captured-scope resolution (manifest scope, legacy inference fallback), idempotency probe, upload, prune, scope-conversion gate (gVisor FULL→DATA Unimplemented, TODO #790), microVM durable-disk extraction; ReasonLocalSnapshotGone. Soft-depends on PR 1.

PR 5 — Enable suspend from PAUSED (the feature)
FromPaused discriminator, MarkSuspendingStep accepts PAUSED + early scope rejections, CallAteletSuspendStep paused branch via DialForNode, functional tests. Depends on PRs 1–4.

Deploy-order note: atelets need PR 4 rolled out before PR 5's ateapi reaches production — an old atelet answers the new RPC with Unimplemented (retryable, but the actor sits in SUSPENDING until the node upgrades).

Deferred follow-ups: gVisor FULL→DATA extraction after #790; janitor/retry budget for a SUSPENDING actor whose snapshot node is permanently gone; updating the docs/architecture.md state diagram (predates pause).

Record the captured scope in the snapshot manifest (#810)

Record the pause capture scope in LocalSnapshotInfo (#812)

Add DialForNode and always finalize in FinalizeSuspendedStep

Add DialForNode and always finalize in FinalizeSuspendedStep (#813)

atelet: add UploadPausedCheckpoint for suspending paused actors

Suspend a PAUSED actor by uploading its local snapshot

Support suspending a PAUSED actor without waking it (#816)

Michelle Au (@msau42)

Answering a review point Michelle Au (@msau42) raised in #660 (comment). It applies to the offload mechanism, which landed here in #816, so recording the analysis on this issue.

We will have some intermediate state where the local snapshot is deleted but the LocalSnapshotInfo hasn't been updated yet. We should think about the failure modes and recovery path.

The window exists but #816 made it recoverable. LocalSnapshotInfo is cleared in workflow_suspend.go:419 at uspend finalization. The paused-origin suspend runs:

ensureMarkedSuspending: mark SUSPENDING (mints and persists InProgressSnapshotName)
ensurePausedSnapshotUploaded: atelet UploadPausedCheckpoint, which uploads the files with the manifest last as the commit marker and then prunes the local dir
ensureVolumesDetached: detach volumes
ensureSuspendedFinalized: finalize (create ActorSnapshot, flip SUSPENDED, clear LocalSnapshotInfo)

So from the prune until finalize, the actor is SUSPENDING with a LocalSnapshotInfo pointing at a node whose directory is already empty.

Failure modes

All recover on a plain re-invocation of SuspendActor, since the destination is minted once and persisted, so every retry writes the same object names:

Before the remote manifest lands: local intact, retry re-uploads and overwrites.
After the manifest, before the prune: both copies exist, retry re-uploads idempotently and prunes.
After the prune, before finalize (the window in question): retry finds no local manifest, probes the remote one, finds it, returns success, finalize completes.
Inside finalize, between CreateActorSnapshot and the status flip: retry re-enters, upload succeeds via the same probe, and CreateActorSnapshot tolerates ErrAlreadyExists.

What's still open

Recovery is entirely client-driven: nothing sweeps a stalled SUSPENDING actor, and it can't be resumed while in that state, so it stays stuck until someone calls SuspendActor again. This is the "janitor/retry budget for a SUSPENDING actor whose snapshot node is permanently gone" follow-up listed as deferred in the implementation plan above.

We still need the janitor to scan and retry suspend for SUSPENDING actors, not scoped to paused-origin actors IIUC.

Tracked in #1502

Navigation Menu

Repository navigation

Support suspending a PAUSED actor without waking it #791

Design summary

PR breakdown

Failure modes

What's still open

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support suspending a PAUSED actor without waking it #791

Description

Activity

dberkov commented on Aug 8, 2026

Design summary

PR breakdown

ILLUM1N0X commented on Aug 12, 2026

Failure modes

What's still open

HavenXia commented on Sep 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions