[Feature] Implementation of Actor State Machine: Lifecycle and Transitions

Issue Description

This issue tracks the formal implementation of the actor state machine within the substrate.

To achieve high density and efficiency, we must distinguish between different "resting" states for actors - specifically focusing on the trade-offs between resume speed, data durability, and resource costs. The implementation must specifically address a potential NIC saturation problem. A simplistic approach of uploading full memory and filesystem images over the network during every suspend/resume cycle would quickly overwhelm the host VM's network interface card (NIC). Consequently, the state machine implementation must prioritize keeping data on the local host to maintain required performance and density.

The "story" of an actor follows a journey from active execution through various stages of dormancy or failure, driven by both user actions and system timeouts.

Actor State Definitions

State	Description	Durability & Performance
RUNNING	The actor is currently active and executing on a worker.	Active CPU/RAM usage.
PAUSED	The actor is offloaded from the CPU, with snapshots stored locally on the node VM.	Fast resume; lower durability (data lost if node fails); cheaper (local, already provisioned storage, no NIC); we can optimize this (e.g., save to peers opportunistically (TODO add issue link)).
SUSPENDED	The actor is offloaded from the CPU, and snapshots are persisted to durable storage.	Higher SLO/durability (e.g., saved to blob storage); expensive (external storage and NIC); requires retention policies; slower resume due to data transfer; can be symbolically tagged.
CRASHED	A failure state reached if the host node fails while an actor is Running or Paused.	Requires manual recovery or revert to a known good state.

Snapshot layers

---
config:
  treemap:
    showValues: false
---
treemap-beta
"Actor's process"
    "memory" : 40
    "local storage"
        "rootfs"
            "durabledir" : 20
            "oci bundle" : 80:::noborder

classDef noborder stroke-width:0px;

memory - a snapshot of an actor's process memory. A snapshot cannot be used for restoration if the OCI image of the actor has been changed (e.g., upgraded).
rootfs - an entire boot disk snapshot with all files. A snapshot cannot be used for restoration if the OCI image of the actor has been changed (e.g., upgraded).
durabledir - a durable directory on rootfs that persists across resumes and participates in snapshots. Can be used for restoration even if the OCI image has been changed.

Actor Lifecycle Story

---
title: Actor State (`ate actor ...`)
---
stateDiagram-v2
direction LR

classDef crushedEvent fill:#f00,font-weight:bold,stroke-width:2px

[*] --> Suspended : create --from

Suspended --> Running : resume
Suspended --> Suspended : commit --tag | revert --to
Suspended --> [*] : delete

Running --> Running : commit --remain [--tag]
Running --> Paused : pause [--timer]
Running --> Suspended : commit [--tag] | revert --to
Running --> Crashed : [VM crashed | ateom crashed | etc...]

Paused --> Running : resume 
Paused --> Suspended : commit [--tag] | revert --to | (auto-commit-timeout)
Paused --> Paused : commit --remain [--tag]
Paused --> Crashed  : [VM crashed | ateom crashed | etc...]
Paused --> [*] : delete

Crashed --> Suspended : revert --to | dump | commit --tag
Crashed --> [*] : delete

class Crashed crushedEvent
class end crushedEvent

1. Create a new actor

Action: ate actor create [--from]
Mechanism: The system creates a new actor from either the default configuration or a tagged snapshot.
Transition: NONE → SUSPENDED

2. Active Execution to Temporary Pause

When a user needs to temporarily free up vCPU resources but expects a quick return, the actor moves to a PAUSED state.

Action: ate actor pause [--timer]
Mechanism: The system takes a snapshot and keeps it on the local node. If an actor remains paused for an extended period and --timer is provided, the system escalates it to a SUSPENDED state to ensure data safety and free up local node storage.
Transition: RUNNING → PAUSED

3. Active Execution to Long-term Suspension

When a user needs to free up vCPU resources and does not expect a quick return, or needs to ensure data safety.

Action: ate actor commit [--tag]
Mechanism: The system takes a snapshot and moves it to durable, persistent storage. A tag can be attached to the snapshot state, allowing the future creation of a new actor from this state (fork) or reverting the current actor to this state.
Transition: RUNNING → SUSPENDED

4. Resuming Operations

Actors can be brought back to a RUNNING state from either dormancy stage.

Action: ate actor resume
From Paused: The scheduler prefers the original node to avoid data transfer.
From Suspended: The system copies the latest snapshots to a node where a new available worker is running.
Transition: PAUSED/SUSPENDED → RUNNING

5. Handling Failures

If the underlying node fails or ateom crashes while the actor's state is only stored locally (Running or Paused), the actor enters the CRASHED state.

Recovery: Users can use ate actor revert --tag to abandon the lost local data and resume from the last known durable snapshot.
Debugging: A user can run ate actor dump to transition an actor to the SUSPENDED state by dumping the current in-memory state while the node VM is still alive, allowing for future debugging.
Transition: RUNNING/PAUSED → CRASHED

Actor’s Storage Snapshots configuration

snapshotsConfig:
  onPause: (full | data)
  onCommit: (full | data)

The substrate supports two snapshot scopes:

full - captures process memory plus the entire filesystem delta on top of the OCI image (including any attached durabledir volumes).
data - captures only the contents of attached volumes that support snapshots (currently durabledir-typed volumes). Process memory and the rest of rootfs are excluded.

onCommit must be a subset of onPause. Concretely: if onPause is full, then onCommit can be full or data; if onPause is data, then onCommit must be data.

Specific system events necessitate the discarding of certain data layers - a process termed "devolving." For instance, updating an actor's OCI image invalidates both rootfs and memory, whereas a gVisor version update permits rootfs reuse while requiring a memory purge.

data:
- Use Case: Suitable for fast-starting agents capable of autonomous state recovery.
- Persistence: Persists only durabledir volumes; designed to survive OCI image upgrades and gVisor changes.
full:
- Mechanism: Comprehensive snapshot of process memory plus the entire filesystem delta (including durabledir volumes).
- Use Case: Critical for slow-starting actors with significant RAM-resident state (e.g., actors with slow bootstrap entry point and/or actors with state in RAM).
- Devolution Path: Reverts to data scope on hardware/gVisor changes or OCI image upgrades.

Failure Modes

SUSPENDED State Details

Resume: If the actor snapshot is missing, the system transitions it to the CRASHED state. Should the snapshot fail checksum validation or prove unusable, the actor moves to CRASHED.
Tagging: Existing tags trigger an error during assignment unless the -f flag is employed.
Revert Operations: Attempts to revert to a non-existent commit or tag result in a system error.

RUNNING State Details

Asynchronous Events (Containers & Kernel): Internal container failures force a transition to CRASHED, though the system attempts to preserve the rootfs and durabledir. A gVisor kernel failure results in a CRASHED state without any data preservation.
Asynchronous Events (Workers): If the worker (ateom) crashes, the actor moves to CRASHED while saving the essential filesystem state, regardless of recovery speed. Permanent worker loss triggers the CRASHED state with metadata preservation. API-driven deletions put workers in a "deleting" phase until all actors are cleared, mirroring the autoscaler's scale-down logic.
Asynchronous Events (Host Nodes): Host node failures lead to a CRASHED state; filesystem data is retained for future recovery. Prolonged node downtime necessitates a transition to CRASHED, saving the current rootfs. Should a node never return, a controller is required to finalize the move to CRASHED.
Commit: Failures during snapshotting return an error, leaving the actor RUNNING. Persistent storage upload errors maintain the RUNNING state, though critical failures may result in a CRASHED status.
Revert To: Errors are thrown if the target commit identifier or tag is missing.
Pause: If snapshot creation fails, the actor remains in the RUNNING state and reports an error.

PAUSED State Details

Asynchronous Events: Node crashes result in a transition to CRASHED; however, since the state was local, it remains preserved if the node returns. Extended node failures follow the same CRASHED logic with the pre-saved local state. Total node loss requires controller intervention to move the actor to the CRASHED state.
Commit: If the underlying local data is missing, the actor enters the CRASHED state. Checksum failures on local data also trigger a transition to CRASHED. Persistent storage upload failures return an API error while maintaining the PAUSED status.
Revert To: If the target tag is invalid, an API error is returned and the state remains PAUSED.
Resume: Actor data missing from local storage results in a move to CRASHED. Corruption detected via checksum leads to a CRASHED state.

CRASHED Recovery Options

Snapshot Dump: Execute ate actor dump <aid> <tag> to persist filesystem state (excluding memory) to a new tag; transitions to SUSPENDED at the prior commit.
State Commitment: Use ate actor commit <aid> to push the current head state to SUSPENDED.
Reversion: Running ate actor revert abandons the failed local data and returns the actor to a SUSPENDED state based on the previous stable commit.

So, we are committed to the noun-first restructure of the CLI UX? Let's call that out explicitly please.

Some bits here seem to be gvisor specific, separable homedir/rootfs layers etc. true?

Davanum Srinivas (@dims)

Davanum Srinivas (@dims)

Some bits here seem to be gvisor specific, separable homedir/rootfs layers etc. true?

Could you please clarify what do you mean by gVisor specific. Both microVM & gVisor have memory, rootfs and homedir. The proposal for homedir snapshot, is as part of the resume, use the clean rootfs, override the homedir with content of snapshot and start bootstrap of the entry point.

Dmitry Berkovich (@dberkov)

Dmitry Berkovich (@dberkov) a couple of statements made me pause to ask the question:

whereas a gVisor version update permits rootfs
Reverts to rootfs + homedir on hardware/gVisor changes

There wasn't a reference to microvm ( https://www.qemu.org/docs/master/system/i386/microvm.html ) either.

As long as gvisor works with one additional alternative as well, i feel comfortable. thanks!

Dmitry Berkovich (@dberkov)

Dmitry Berkovich (@dberkov) looking deeper a couple of other things stood out

`ate actor dump` has two contradictory prose?

Which one is true? "the current in-memory state" or "filesystem state (excluding memory)."

A user can run ate actor dump to transition an actor to the SUSPENDED state by dumping the current in-memory state
Execute ate actor dump <aid> <tag> to persist filesystem state (excluding memory)

Is `SUSPENDED` → `CRASHED` is missing from the diagram?

the Failure Modes → SUSPENDED section explicitly routes failed resumes there.

revert always demotes to SUSPENDED

revert --to v1 from RUNNING throws away your live process and drops you to cold durable storage — so you then have to resume (another transfer) to use the state you reverted to?

commit means three unrelated operations

RUNNING / PAUSED: snapshot live or local state → durable (real work, uses NIC).
SUSPENDED commit --tag: the actor is already durable and not running, so there's nothing to snapshot — this is purely attach a tag (metadata).
CRASHED commit: "push the current head to durable," which is nearly identical to revert --to .

This should be considered a proposal, not a set-in-stone design, but we need to drive fast on this one.

I marked this P0 since I think we need to do a minimal impl ASAP to prove that it works. It really rests on a few assumptions:

"local storage" exists and is meaningful (not often true on cloud VMs)
"local storage" is actually faster / less expensive than networked blob storage
We can do effective management of local snapshots (treat as a cache)

The state diagram shows a lot of edges not detailed here. That's OK, but as we get to implementation we should sort which edges are the most important and which can come later.

So, we are committed to the noun-first restructure of the CLI UX? Let's call that out explicitly please.

I think those are orthogonal, this doc needs to express the ideas SOMEHOW. IMO noun-first makes more sense for these, otherwise we end up with "ate commit" and "ate pause" and "ate resume" and "ate revert" and "ate dump" -- not terrible, but it pollutes the "top level" CLI namespace.

Some bits here seem to be gvisor specific, separable homedir/rootfs layers etc. true?

It should not be gvisor specific. I'm presuming any implementation of an ateom should be able to mount a directory/volume distinct from the rootfs -- is that not true for some impl?

As long as gvisor works with one additional alternative as well, i feel comfortable. thanks!

ACK - we need a microVM impl ASAP to keep us all honest.

ate actor dump has two contradictory prose?
Which one is true? "the current in-memory state" or "filesystem state (excluding memory)."

A user can run ate actor dump to transition an actor to the SUSPENDED state by dumping the current in-memory state

Execute ate actor dump to persist filesystem state (excluding memory)

I think dump should mean "as much as possible". If the machine has died and the state is lost, all we can do is revert back to some previous commit. If the process segfaulted, there's no useful memory snapshot, but there SHOULD BE rootfs and homedir.

If you buy the snapshotConfig part, here's how I think of it. none < homedir < process. The onCommit and onPause configs can be different, but onCommit is always <= onPause. So concretely: You can set onPause: process and onCommit: homedir, meaning we store the larger full-process snapshot AND homedir when pausing locally, but only the homedir when committing to permanent storage (which makes scheduling back to the same node very important). IN the context of dump it means "recover as much as possible, up to the onCommit config". If successful, that can then be used (I guess via revert, which is a little weird?).

revert always demotes to SUSPENDED

Yeah, this is a little unusual, but I think we should see if RUNNING --revert--> RUNNING really needed -- it seems like it will add complexity that will rarely be needed.

commit means three unrelated operations

RUNNING / PAUSED: snapshot live or local state → durable (real work, uses NIC).

SUSPENDED commit --tag: the actor is already durable and not running, so there's nothing to snapshot — this is purely attach a tag (metadata).

CRASHED commit: "push the current head to durable," which is nearly identical to revert --to .

Logically we have two operations - "commit" and "tag" which sometimes are used together and sometimes not. commit [--tag] means you can think of the combination as an atomic operation. We could describe it as ate commit and ate tag (non-atomic) or we could describe ate tag and ate commit [--tag] and 2 ways of achieving the same result. I think there's value in having less verbs, but not if it is confusing. Note, we haven't describe this at the API layer yet. To me (and I am biased), commit --tag makes sense -- it means "commit any dirty state (including none) and add a tag". Calling commit on a SUSPENDED state should be a no-op (there's no dirty state, by definition). I am happy to be argued down -- I don't feel strongly.

The CRASHED -> SUSPENDED edge is genuinely weird. I am uncomfortable (for now) being too opinionated about error recovery, and I think we should leave it to the caller (for now). At this layer of a system, we don't know what's going on inside, so some folks will want to do quarantine and post-mortem, some will want to just try again.

We could do CRASHED --resume--> RUNNING, but it's not clear to me if that is useful. Heck we could make that automatic, but it's not obviously correct. Forcing CRASHED to go back to SUSPENDED seems like the baseline requirement, and then we can explore if we need richer handling.

But here's where it is tricky, IMO:

Think of a given actor as a time-series of N committed snapshots (where N is governed by some retention policy). Any commit can be tagged, and tagged commits are subject to different retention policies. At any time you can "fork" an actor into a new actor with create --from=<actor:tagname> or "revert" an actor to a previous state with revert (RUNNING->SUSPENDED) or revert --to=<tag>. There's an implicit "HEAD" or "latest" tag.

But again, CRASHED is weird.

I want to be able to save the CRASHED state, but MAYBE not keep it. If commit pushes the crashed state into the sequence as HEAD, retention policy might discard the previous state. We could either:
a) Force the caller to create a temporary tag for HEAD, commit the CRASHED state, inspect it, then decide if they should keep it or revert --to <tmp>, then finally delete the tmp tag
b) Force the caller to commit the CRASHED state to a new tag which DOES NOT replace HEAD, inspect it, then decide if they should keep it (revert --to <crashed>) or delete the tag and let the commit get GC'ed

The current diagram describes (b) but I could see some variation of (a). Input welcome.

This should be considered a proposal, not a set-in-stone design, but we need to drive fast on this one.

Sounds good. Just pointing out things as we go along.

I hit the button too soon. Updated the comment - sorry for the book.

I don't like the generated diagram, here's my manual version (from https://docs.google.com/drawings/d/13ZFdJhmOfX1RaNInjMfs6xjzmPQlmY5BiBEPgV0lFJY/edit which is shared with ate-dev mailing list since I can't share to "world" on corp account :)

Dmitry Berkovich (@dberkov)

Dmitry Berkovich (@dberkov) Tim Hockin (@thockin) Reading back over this when reviewing #295, are we doing separate:

process memory ("process")
homedir ("homedir")
process memory + full rootfs (?? "full" ?? naming is hard ...)

Or just the last two?

This impacts how we handle rootfs for uVM, I may need to switch to one of the other approaches prototyped so far, I was aiming to resolve 1) which is tricky in uVM.

There's also some mention of "none", but that's basically just "always create from golden snapshot", we don't need to alter how snapshotting works for that one.

I do think realistically you have to configure these up front in the ActorTemplate, because we need to alter how the sandbox is spawned (e.g. for full we might use tmpfs and just preserve memory as we do today in gvisor, but for a pure "process" mode we'd want to skip the tmpfs upper and write to disk + discard it).

Benjamin Elder (@BenTheElder)

Benjamin Elder (@BenTheElder)

are we doing separate:
process memory ("process")
homedir ("homedir")
process memory + full rootfs (?? "full" ?? naming is hard ...)

Just last two. While designing the feature, I did not find any reasonable use-case to take process snapshot only and no disk.

In #287 I have not implement "NONE", since it does not make sense. Like what a point of suspending actor and not taking a snapshot? It is like delete actor and create a new one.

I will update the design doc to exclude "none" from it.

Just last two. While designing the feature, I did not find any reasonable use-case to take process snapshot only and no disk.

ACK.

I feel like we will actually need the opposite of homedir then: Some way to mount truly disposable scratch path which will not be preserved / will not bloat snapshots.

If we offer that, it can't be the typical tmpfs since we're storing memory in both gvisor and uVM process snapshots, but we can mount a scratch disk volume. For uVM that must not be virtiofsd (pass through from the host) because the caches are in memory, we can mount a disposable block device.

Like what a point of suspending actor and not taking a snapshot? It is like delete actor and create a new one.

I'm imagining we could allow you to do some initialization (say loading python libraries) in the golden snapshot process and then repeatedly spawn the golden snapshot to exec (with some output over the network)? I think it's pretty easy to revisit "none" later anyhow though. Something like Golden => Resume => Delete.

DurableDir support (#295)

Detect `CRASHED` Actors during Checkpoint and Restore (#353)

I think maybe we should change how we expose suspend vs pause #798

Support suspending a PAUSED actor without waking it (#816)

Navigation Menu

Repository navigation

[Feature] Implementation of Actor State Machine: Lifecycle and Transitions #119

Issue Description

Actor State Definitions

Snapshot layers

Actor Lifecycle Story

1. Create a new actor

2. Active Execution to Temporary Pause

3. Active Execution to Long-term Suspension

4. Resuming Operations

5. Handling Failures

Actor’s Storage Snapshots configuration

Failure Modes

SUSPENDED State Details

RUNNING State Details

PAUSED State Details

CRASHED Recovery Options

`ate actor dump` has two contradictory prose?

Is `SUSPENDED` → `CRASHED` is missing from the diagram?

revert always demotes to SUSPENDED

commit means three unrelated operations

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature] Implementation of Actor State Machine: Lifecycle and Transitions #119

Description

Issue Description

Actor State Definitions

Snapshot layers

Actor Lifecycle Story

1. Create a new actor

2. Active Execution to Temporary Pause

3. Active Execution to Long-term Suspension

4. Resuming Operations

5. Handling Failures

Actor’s Storage Snapshots configuration

Failure Modes

SUSPENDED State Details

RUNNING State Details

PAUSED State Details

CRASHED Recovery Options

Activity

dims commented on May 29, 2026

dberkov commented on May 29, 2026

dims commented on May 29, 2026

dims commented on May 29, 2026

ate actor dump has two contradictory prose?

Is SUSPENDED → CRASHED is missing from the diagram?

revert always demotes to SUSPENDED

commit means three unrelated operations

thockin commented on May 30, 2026

dims commented on May 30, 2026

thockin commented on May 30, 2026

thockin commented on May 30, 2026

15 remaining items

BenTheElder commented on Jun 24, 2026

dberkov commented on Jun 24, 2026

BenTheElder commented on Jun 24, 2026

BenTheElder commented on Aug 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`ate actor dump` has two contradictory prose?

Is `SUSPENDED` → `CRASHED` is missing from the diagram?