Skip to content

Handling Resume from Pause failures #660

New issue
New issue
Closed
Feature
#908
Closed
Handling Resume from Pause failures#660
Feature
#908
Assignees
ILLUM1N0X
Labels
area/storagekind/featureAn enhancement / feature request or implementation
Milestone
M2

Description

@msau42
Michelle Au (msau42)
opened on Jul 31, 2026

When we pause an actor, the actor is forced to resume on the same node. However, at the time of resume, the node may not have capacity for the actor anymore. Let's discuss how we can recover from this situation in the near term.

My thoughts for the desired near term flow:

  1. Resume() returns an out of capacity error
  2. User needs to trigger Suspend() in order to offload the snapshot to remote storage.
  3. User can then retry Resume()

In the future, we could improve on this by automatically doing steps 2+3 as part of the initial Resume() call.

Activity

  1. Michelle Au (msau42)
    added
    area/storage
    on Jul 31, 2026
  2. Michelle Au (msau42)
    added the
    Feature
    issue type on Jul 31, 2026
  3. Michelle Au (msau42)
    added this to the M2 milestone on Jul 31, 2026
  4. Zheng Lammerts (zlammerts-svg)
    added this to Substrate Milestones (go/substrate-milestones)on Jul 31, 2026
  5. github-project-automation
    moved this to Todo in Substrate Milestones (go/substrate-milestones)on Jul 31, 2026

  6. ILLUM1N0X commented on Aug 3, 2026

    @ILLUM1N0X
    ILLUM1N0X
    on Aug 3, 2026
    Collaborator

    Implementation Plan: Near-Term Flow for Resume from Pause Failures

    To recover from out-of-capacity failures when resuming an Actor from STATUS_PAUSED, we will implement the near-term flow (Pause -> out of capacity -> Suspend -> Resume on any node).

    Key Design Decisions

    • Explicit Error Signaling: Resume() returns codes.ResourceExhausted when a node lacks capacity, avoiding silent cloud storage uploads or SLA violations.
    • Service Separation (NodeStorage): Paused actors have no active ateom pod. Instead of adding a filesystem operation to AteomHerder, we introduce a dedicated NodeStorage gRPC service on atelet for node-level filesystem offloading.

    Implementation Phases

    1. Define NodeStorage Service (internal/proto/ateletpb/atelet.proto)
      • Add service NodeStorage with rpc OffloadLocalCheckpoint. Allows ateapi to upload local checkpoint files directly from disk without an active ateom pod.
    2. Implement NodeStorage Server (cmd/atelet/main.go)
      • Implement OffloadLocalCheckpoint to read checkpoint files from ateompath.LocalCheckpointsDir, upload them to remote storage, and delete the local directory.
    3. Support STATUS_PAUSED in Suspend Workflow (cmd/ateapi/internal/controlapi/)
      • dialer.go: Add DialForNode(nodeName string) to AteletDialer (extracting pod lookup/connection logic from DialForWorker) since paused actors have no worker pod name.
      • workflow_suspend.go:
        • Allow actors in STATUS_PAUSED. When paused, call dialer.DialForNode(nodeName) and invoke NodeStorage.OffloadLocalCheckpoint.
        • Clear LocalSnapshotInfo upon entering STATUS_SUSPENDED, removing the node scheduling constraint so Resume() can schedule anywhere.
    4. Standardize Capacity Errors (cmd/ateapi/internal/controlapi/workflow_resume.go)
      • In AssignWorkerStep, return codes.ResourceExhausted ("no free workers available matching scheduling constraints") when scheduler.Schedule() fails due to capacity.
    5. Automated Testing (cmd/ateapi/internal/controlapi/)
      • Add unit tests in workflow_suspend_test.go and an e2e test in functional_test.go verifying: Pause -> capacity failure -> Suspend -> relocation Resume.

  7. msau42 commented on Aug 6, 2026

    @msau42
    Michelle Au (msau42)
    on Aug 6, 2026
    CollaboratorAuthor

    Overall approach lgtm. Just a few comments:

    • I'm not sure if we need a separate service. There is some overlap with existing Checkpoint/Resume calls wrt managing local snapshots, and we'll probably need to ensure some concurrency control among all 3 methods. The existing service dials into atelet, not the ateom pod.
    • We will have some intermediate state where the local snapshot is deleted but the LocalSnapshotInfo hasn't been updated yet. We should think about the failure modes and recovery path.
    • Also keep in mind atelet: prune local pause snapshots #705 which is also handling the snapshot cleanup

  8. msau42 commented on Aug 7, 2026

    @msau42
    Michelle Au (msau42)
    on Aug 7, 2026
    CollaboratorAuthor

    Related issue: #791

  9. BenTheElder commented on Aug 7, 2026

    @BenTheElder
    Benjamin Elder (BenTheElder)
    on Aug 7, 2026
    Collaborator

    Longer term I think we should consider direct p2p instead of bouncing through GCS, for latency reasons. I also filed #798

  10. ILLUM1N0X commented on Aug 10, 2026

    @ILLUM1N0X
    ILLUM1N0X
    on Aug 10, 2026
    Collaborator

    The offload mechanism (enabling SuspendActor on a PAUSED actor) is tracked in #791 and currently being implemented by Dmitry Berkovich (@dberkov).

    I will follow up by reviewing those PRs. Once they land, I'll work on standardizing the out-of-capacity error in ResumeActor (codes.ResourceExhausted) and adding an end-to-end test for the full pause-to-resume capacity recovery flow.

  11. ILLUM1N0X
    moved this from Todo to In Progress in Substrate Milestones (go/substrate-milestones)on Aug 11, 2026
  12. Bowei Du (bowei)
    assigned
    ILLUM1N0X
    on Aug 11, 2026

  13. ILLUM1N0X commented on Aug 12, 2026

    @ILLUM1N0X
    ILLUM1N0X
    on Aug 12, 2026
    Collaborator

    Phases 1–3 landed under #791, though not as written above (the offload RPC went on the existing AteomHerder rather than a new service). Remaining work is phases 4 and 5, revised below.

    Phase 4 — Unambiguous capacity signal

    Return codes.ResourceExhausted from assignWorkerAttempt for scheduling.ErrNoCapacity (currently returns FailedPrecondition in workflow_resume.go).

    We can't return FailedPrecondition as-is: it is overloaded on this RPC, since it also covers "the actor isn't in a resumable state" (workflow_resume.go) and the two demand opposite client responses.

    We considered keeping FailedPrecondition and carrying an AIP-193 ErrorInfo reason to disambiguate, which would have left the router untouched. It doesn't work: ateapi's ServerUnaryInterceptor rebuilds every error as status.Error(code, message) and drops details.

    FailedPrecondition is currently the de-facto capacity signal, so three consumers move with it:

    1. The router's park predicate (resumer.go): add ResourceExhausted so parking is unchanged.
    2. The HTTP mapping (errors.go): fold ResourceExhausted into the FailedPrecondition/Aborted branch (503 + description). Its current "actor %s rate limited" message was written speculatively since nothing in substrate produced the code until now.
    3. The no_capacity metric bucket (extproc/metrics.go): move it to the new code.

    Accepted risk: we don't exclusively own ResourceExhausted (grpc-go emits it on message-size violations), so a passthrough would be parked once before failing. Not reachable in practice given the payload sizes on this path.

    Phase 5 — Relocation test

    The test goes in functional_test.go with a synthetic two-node fleet, building on the harness TestSuspendActor_FromPaused already uses.

    TestResumeActor_RelocatesAfterSuspendFromPaused:

    1. createTemplate + createWorkerPod(..., "worker-1", "node1", "pool1").
    2. Create actor A, resume (lands on worker-1), pause → PAUSED, pinned to node1, worker-1 freed.
    3. Create actor B and resume it → B occupies worker-1, so node1 has no free worker.
    4. createWorkerPod(..., "worker-2", "node2", "pool1") → free capacity exists, but on the wrong node.
    5. ResumeActor(A) → assert ResourceExhausted.
    6. SuspendActor(A) → assert SUSPENDED and LocalSnapshotInfo == nil.
    7. ResumeActor(A) → assert it succeeds and the worker assignment is worker-2 on node2.
  14. Aditya Shantanu (aditya-shantanu)
    added
    kind/featureAn enhancement / feature request or implementation
    on Aug 12, 2026
  15. Haven Xia (HavenXia)
    closed this as completedin #908on Aug 17, 2026
  16. github-project-automation
    moved this from In Progress to Done in Substrate Milestones (go/substrate-milestones)on Aug 17, 2026
  17. Mayowa Fajobi (MayorFaj)
    added a commit that references this issue on Aug 17, 2026

    ateapi: return ResourceExhausted when no worker is free (agent-substr…

    2af5308
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/storagekind/featureAn enhancement / feature request or implementation

Type

Feature

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions