Add the CRASHED state to Actor State Machine

Description: Discover all scenarios where actors are in a state where manual recovery is required.
#119 has listed some examples. Implement the transition to the CRASHED state.

CRASHED means the actor can not be automatically recovered, and requires manual intervention to revert to a known good state.

Out of the transitions to CRASHED state mentioned in #119, I think we have 2 categories

Ate system detects a not-automatically recoverable error during Chekpoint/Restore operations. For example, we detect missing or corrupted snapshots during checkpoint or restore. - We have entry points in the codebase that we can add the curr_state -> CRASHED transition.
Atelet or ateom crashes or becomes unhealthy. This will impact any actors that are in the RUNNING or PAUSED state, but not actors in other states. - Currently we don't watch atelet or ateom health, so this could go undetected.

In the second category, if atelet itself is unreachable it's possible that the actor and ateom is still running, only atelet is unhealthy.

Related to snapshotConfig - I think the snapshotConfig is what the user intended be snapshotted (none | homedir | process), but when the Actor is crashed, it's possible we can partially snapshot something (for example, if the actor OOM'd, we can still take snapshot of homedir) so we could save an additional actualSnapshotted for recovery?

Next step: Create a failure matrix with the following columns:

Actor starting state: Running|Paused etc.
Failure reason: eg. Actor OOM? Ateom crash??
End state: CRASHED
Detected by: Ateom ?
Recoverable layers:
- memory: ?
- homedir: ?
- rootfs: ?
Possible recovery steps:
1. [Substrate] snapshot filesystem state (excluding memory) to a new tag, do not delete the previous snapshot.
2. [AgentDev] run ate actor commit actor_id to revert to revert to a chosen snapshot.

I think CRASHED could be detected by Ateom, but you might as well have Ateom crash itself on error and have the detection in Atelet. Atelet will have to handle Ateom crashing anyway?

Bowei Du (@bowei)

Thanks Bowei Du (@bowei) ! This will simplify the Ateom logic. So when an Actor crashed, we can have Ateom:

Do some work to salvage what is on the disk, (eg. when the customer's snapshotConfig is "homedir only")
Crash itself, and report to atelet what has been saved, maybe on the next heartbeat?

Category 1: Actor crash detected during ateapi Checkpoint/Restore operations

ResumeActor:
1. During ResumeActor we'll put the actor in CRASHED state if the Resume failed due to any errors related to the snapshot itself, which includes:
  1. Missing or corrupted snapshot when downloading from GCS
  2. Runsc command fails.
2. All other types of errors will not put the Actor in CRASHED state.
SuspendActor or PauseActor
1. During SuspendActor or PauseActor we'll put the actor in CRASHED state if we cannot produce a valid snapshot, which includes:
  1. cannot connect to Ateom after retries,
  2. runsc command fails.
  3. cannot upload the snapshot to GCS (during suspend) ; or run out of disk space locally (during pause).
2. All other types of errors will not put the Actor in CRASHED state.

Category 2: Actor crash detected while the Actor is Running or Paused, outside of ateapi operations

WIP

Question about 1.ii -- if we get an error that we didn't know about, this should always result in CRASHED state? Unless it's on a well-known list of errors we can recover from, how would the software know how to proceed on an arbitrary error? Example: we ran out of disk space or some component timed out or we got bad RAM.

During ResumeActor, if we fail due to out of disk space or bad RAM, I still think of the Actor as not CRASHED because

A substrate user can retry ResumeActor on a different node or worker
if a substrate user runs kubectl ate actor commit <> to a different snapshot, it wouldn't fix this. (diagram for state transition is in [Feature] Implementation of Actor State Machine: Lifecycle and Transitions #119 (comment))

iow I thought of the Actor as existing only as a "snapshot" when it's not running, so it's CRASHED only if the snapshot is corrupted. If the node has hardware failures, I think we can mark the node as unhealthy to prevent actors from resuming on it.

1.i.b Runsc command fails.

Is every runc failure supposed to trigger CRASH? We might can ask from gVisor team to separate retryable vs non-retryable failure.
Same about CHV too.

Detect `CRASHED` Actors during Checkpoint and Restore (#353)

Adding live production-shaped evidence (GKE, 2026-07-30 chaos/soak campaign) that the missing recovery path here is the single biggest durability gap we measured — and a case for prioritizing the revert edge that #119 already designs (Crashed --> Suspended : revert).

What we measured: after one night of routine-fault injection (hourly worker-pod kills — i.e., what a node upgrade or pool resize does), 6 of 8 test actors were STATUS_CRASHED. Every one of them had a fully valid last-committed snapshot in the bucket (~1,500 committed generations each) and durable state accumulated over thousands of cycles. None of it is reachable:

ResumeActor → FailedPrecondition (workflow_resume.go:141-145 accepts only SUSPENDED/PAUSED)
UpdateActor only mutates WorkerSelector — status not writable
no restore/revert verb in ateapi.proto
only exit is DeleteActor; recreate starts from the golden snapshot → all state since creation lost

So for the dominant crash causes (worker pod deleted, node drained — not workload corruption), the system durably holds the data and refuses to serve it. Notably #362/PR #353 assume a crash implies the latest snapshot is lost; for these causes the snapshot is intact.

Related threads to reconcile when picking this up:

Add first-class ActorSnapshot lifecycle APIs #529 currently lists "in-place rollback of an existing Actor" as a non-goal — implementing this edge probably means promoting it, or shaping it as ResumeActor-accepts-CRASHED-when-a-committed-manifest-exists (documented as losing post-snapshot state).
Open PR Document actor lifecycle and sandbox RPC semantics #509's proto docstring would codify today's behavior as intended ("failed irrecoverably … can be inspected or deleted") — worth softening if recovery is planned.
Handle Pod Eviction/Deletion gracefully #23/Handle graceful worker termination #517 reduce how often actors get crashed by infra events, but don't help the ones that do.

Clarify the above, by unknown error -- I really do mean unknown error that we can't know for sure the state is not corrupted.

Agreed, I think we should start with crash the actor on unknown errors types during Chekpoint/Restore, and if we discover new, exempt-able failures, we can exempt them in the future.

For M2, we also need to know if a running Actor has crashed. Ateom needs to detect if the Actor container inside is healthy, and Ate API needs to mark the Actors as Crashed.

I started to test this locally, and #1473 implements the ateom half of category 2, but detection is via atecontroller instead of atelet, so wanted to ask what do we all think about it.

One common failure is getting oomkilled, which is a sigkill and ateom has no chance to crash cleanly, salvage, or report. atecontroller instead compares the container id the kubelet reports against the one recorded on the Worker.

Two notes/questions:

Salvaging in ateom before it crashes itself won't survive an OOM kill. The durable dirs are on the node, so salvage after atelet recovering what is on disk once the ateom is gone might work better.
Here we define CRASHED as "requires manual intervention to revert to a known good state", but we can only delete a CRASHED actor, so one with a good LatestSnapshot can only be thrown away, right? Recycle a worker whose ateom container was replaced under it + telemetry coverage #1473 can send more actors there. Should we track this in a new issue?

I think #1420 is related, the concept of an actor crashing is a nuanced one.

It is both true that an actor can be crashed and that it has salvageable state, and we probably need to represent that more specifically than Crashed: true/false.

Even more confusing: If you asked for full snapshots, we may still be able to recover the rootfs and volumes. But depending on the actor that may be an inconsistent and undesirable state.

I think we should not solve all of those today, but leave room to extend this sort of "something is bad but it's not totally lost" later.

Agreed, but #1473 shouldn't conflict with that. WORKER_SANDBOX_GONE is an ate.failure.reason next to state, not a new state, so a richer "bad but not totally lost" layer can be added on top of that.

A recycle keeps latest_snapshot and local_snapshot_info. It only clears the in-progress checkpoint names, which were half-written anyway.

There's one gap though, as after a crash we no longer know which node a RUNNING actor was on. If "recoverable layers" is going to work, we'd need to save that node somewhere.

Is atecontroller the right place to detect this, or should it move to atelet later? Happy either way.

Crash actor when Ateom Checkpoint or Restore fails; Also when the err…

Navigation Menu

Repository navigation

Add the CRASHED state to Actor State Machine #292

Category 1: Actor crash detected during ateapi Checkpoint/Restore operations

Category 2: Actor crash detected while the Actor is Running or Paused, outside of ateapi operations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add the CRASHED state to Actor State Machine #292

Description

Activity

zoez7 commented on Jun 24, 2026

zoez7 commented on Jun 25, 2026

zoez7 commented on Jun 26, 2026

bowei commented on Jun 26, 2026

zoez7 commented on Jun 26, 2026

zoez7 commented on Jun 29, 2026

Category 1: Actor crash detected during ateapi Checkpoint/Restore operations

Category 2: Actor crash detected while the Actor is Running or Paused, outside of ateapi operations

bowei commented on Jun 29, 2026

zoez7 commented on Jun 29, 2026

dberkov commented on Jun 30, 2026

aditya-shantanu commented on Jul 30, 2026

bowei commented on Jul 30, 2026

zoez7 commented on Aug 10, 2026

krisztianfekete commented on Sep 4, 2026

BenTheElder commented on Sep 4, 2026

krisztianfekete commented on Sep 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions