Actor lifecycle hooks

This issue tracks the design and implementation of actor lifecycle hooks.

These are analogous to Kubernetes container lifecycle hooks. At a minimum we should support 2 new hooks (names TDB).

PostResume - Triggered right after an actor is resumed.
PreSuspend - Triggered right before an actor is suspended.

This area needs more investigation to determine exactly how these hooks should be implemented.

I'm also definitely interested in this, but there's a bit of a wrinkle here i think we need to account for in this design. Specifically, at what point in the snapshot lifecycle does this happen. Before/after makes a very large difference for pre-warming etc.

We are also interested in this.
We would need lifecycle hooks for running commands post-resume + pre-suspend for cuda-checkpointing GPU actors.
This connects to #185.

docs(observability): add actor application telemetry best practices

Scott Funkenhauser (@sfunkenhauser)

Scott Funkenhauser (@sfunkenhauser) I can take this one. The upgrade workstream (#473) has to define these semantics anyway: a blue/green model that batch-suspends every running actor, so the PreSuspend and PostResume are inevitable path on the upgrade.

docs(observability): mark actor telemetry guidance as interim

sfunkenhauser I can take this one. The upgrade workstream (#473) has to define these semantics anyway: a blue/green model that batch-suspends every running actor, so the PreSuspend and PostResume are inevitable path on the upgrade.

Is this required for the initial release? Or is it sufficient to push this responsibility to the actor? The actor receives a SIGTERM when the pod is deleted and decides how to proceed. This was the assumption I was proceeding with for graceful termination, and it seems like upgrades would be very similar?

Or is it sufficient to push this responsibility to the actor? The actor receives a SIGTERM when the pod is deleted and decides how to proceed.

I think SIGTERM only exists when termination happens, right? On the suspend path there's no signals to actors, and according to #503 it's a good addition for in memory states.

"required for the initial release"

I agree, it is not for M2. We can implement post M2.

This was the assumption I was proceeding with for graceful termination, and it seems like upgrades would be very similar?

I think it's similar at the per-actor step, but opposite in the property that matters. Graceful termination(your #517) ends the actor with SIGTERM. But the upgrade drain I am designing wants every actor checkpointed and resumed on the new stack. So the upgrade batch suspends everything proactively, before any pod is deleted - by the time the old pods die they should already be empty, and your SIGTERM path is the safety net for outliers when we turn down the old pods.

Overall I think we can keep this open but note it's a post M2 task. WDYT?

Yes sounds good.

demos: add actor telemetry continuity reference demo (agent-substrate…

Some initial thoughs.

Actor lifecycle hooks: `preCheckpoint` and `postRestore`

Problem

Actors are checkpointed and restored many times over a single process lifetime. Memory survives the boundary; almost nothing else does.

After a restore, the wall clock has jumped, TCP connections are dead on both ends, tokens and leases have expired, timers did not fire. Nothing tells the actor this happened.
Before a checkpoint, the actor gets no warning. A DATA-scope checkpoint captures only the durable-dir volumes, read from the host directory. Writes still sitting in the application's buffers, or in the sandbox's page cache, are silently missing from the snapshot.

I propose two optional HTTP hooks: one that tells the actor a checkpoint is coming so it can get ready, and one that tells it a restore has happened so it can reconcile.

Design

API

One optional field on Container, alongside readyz:

message Lifecycle {
  // pre_checkpoint is called before the actor's state is captured, on both
  // Suspend and Pause.
  //
  // +k8s:optional
  HTTPGetAction pre_checkpoint = 1;

  // post_restore is called after the actor is restored from a snapshot. It is
  // not called on a cold boot.
  //
  // +k8s:optional
  HTTPGetAction post_restore = 2;
}

The hooks are HTTP GET, reusing readyz's HTTPGetAction, with no request body. The ateom calls them over the interior veth, just as it polls readyz. There is no exec or sleep handler for now, but we could consider adding them in the future.

`preCheckpoint`

preCheckpoint signals that a checkpoint is coming shortly. The actor should get ready for it: close open connections (including long-lived streams such as WebSockets), flush buffers to durable-dir volumes, drop state that should not be captured.

It is not an opportunity to finish the current turn. The hook has a hardcoded timeout of roughly 5s, and a turn can take minutes. Deciding when an actor is at a good point to be suspended is the job of an outer loop, such as the idleness probe or whoever calls SuspendActor.

preCheckpoint → deactivate networking → pause / checkpoint

The hook runs before networking is deactivated, so the actor can still flush over the network. Requests still in flight when networking is deactivated are canceled.

`postRestore`

postRestore signals that the actor was restored from a snapshot, so the actor can reconcile: resync the clock, reconnect, refresh tokens.

restore sandbox → activate networking → { postRestore ∥ readyz } → return

It runs in parallel with readyz, so it costs max(hook, readyz) instead of the sum. The restore waits for both, and no traffic is routed to the actor until then. Because the hook sits on the activation path, it has a hardcoded timeout of 500ms.

Behavior

Hooks are fail-open. If a hook returns non-2xx, times out, or cannot be reached, the failure is logged and the checkpoint or restore proceeds. A hook is a signal, not a chance to cancel the suspend or block the restore.

New requests are held off during preCheckpoint. Once the workflow commits SUSPENDING or PAUSING, the router parks new requests for the actor instead of routing them to it. Streams opened before that point (WebSockets, CONNECT tunnels) are not gated, so the actor must close them itself.

A slow preCheckpoint causes 503s. The router parks a request for at most 5s. That window has to cover the hook, the checkpoint, and the resume that follows, so if preCheckpoint takes too long, requests arriving during the suspend start getting 503 errors.

Hooks must be idempotent. They fire on every checkpoint and restore and are delivered at least once.

No payload. Hooks receive no arguments for now. This can be expanded in the future.

Integrating

Serve the routes you need. The status you return is logged, not acted on:

func (a *app) preCheckpoint(w http.ResponseWriter, r *http.Request) {
    a.closeConnections()
    if err := a.flush(); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    w.WriteHeader(http.StatusOK)
}

func (a *app) postRestore(w http.ResponseWriter, r *http.Request) {
    a.reconnect()
    a.refreshTokens()
    w.WriteHeader(http.StatusOK)
}

Appendix

Why GET, not POST

A hook is a signal, not a command: it tells the actor a checkpoint is coming or a restore has happened, and what the actor does about it is up to the actor. GET fits that. It is idempotent, which matches the at-least-once delivery the hooks already require. It is simple, with no body, content type, or encoding to agree on, so the actor needs nothing beyond a route. It matches Kubernetes' httpGet lifecycle handlers, and it reuses readyz's HTTPGetAction.

Why not fail-closed

Fail-closed hooks would let an actor refuse a checkpoint or a restore. For preCheckpoint, that protects an actor whose flush failed from a DATA snapshot that is missing data. The costs:

The actor controls when its worker is reclaimed. A hook that keeps failing blocks every suspend, and the platform can no longer take the worker back without the actor's consent.
A refused suspend strands the actor. The workflow has already committed SUSPENDING, so the actor stays there, holding its worker and serving nothing, until the suspend is retried and succeeds. Requests parked in the meantime start getting 503s after 5s.
A refused restore does not heal on retry. A retry restores the same snapshot, and a reconcile that failed because an external dependency is down fails again, so the actor is unavailable until the dependency recovers.

Fail-open is the right default. Fail-closed could be offered later as a per-template policy for actors that would rather not be suspended than risk losing data.

Waiting for the current turn

An actor may want a suspend to wait until its current turn finishes. preCheckpoint cannot provide that. While the hook runs the router parks new requests, and a parked request waits at most 5s before getting a 503. A turn can take minutes, so blocking the suspend on it would make the actor unavailable to every new caller for the rest of the turn, while it keeps holding its worker. This is out of scope for the initial lifecycle hooks.

Scott Funkenhauser (@sfunkenhauser)

Thanks for writing this up Scott Funkenhauser (@sfunkenhauser), really excited about this one! A lot of our telemetry flushing suffering (see #503) goes away if this is designed with telemetry in mind, so it might be worth calling that out in the Problem section too?

A few questions from the observability side:

Does postRestore also fire when a new actor first comes up from the golden snapshot? That's the case I'm most interested in, because that's where actors would want to reset their OTel SDK. Relatedly, can we count on /run/ate/actor-* already having the new actor's identity when the hook runs? Otherwise we'd get the template's identity.

For preCheckpoint, I'm guessing most actors will just call ForceFlush there. Could we say explicitly that the export path (egress today, the relay later) stays up until the hook returns? And since 5s is shorter than the OTLP exporter's default 10s timeout, we'll probably want to tell people to ForceFlush, and make sure ateom actually cancels the request if it times out.

Would it be possible to send traceparent with the hook requests? Then whatever the actor does in the hook shows up in the same trace as the suspend or resume. It would make slow resumes a lot easier to debug.

Fail open sounds right, but I'm a bit worried failures will be pretty hard to notice when logged only. What do you think about a hook duration metric and a failure event with the actor's identity? I'm happy to track these in new issues.

Btw, I'm assuming hooks don't fire on delete, so the final flush there still relies on SIGTERM?

Does postRestore also fire when a new actor first comes up from the golden snapshot?

Yes.

Relatedly, can we count on /run/ate/actor-* already having the new actor's identity when the hook runs? Otherwise we'd get the template's identity.

Good question. I haven't looked at this yet, but this sounds reasonable.

For preCheckpoint, I'm guessing most actors will just call ForceFlush there. Could we say explicitly that the export path (egress today, the relay later) stays up until the hook returns? And since 5s is shorter than the OTLP exporter's default 10s timeout, we'll probably want to tell people to ForceFlush, and make sure ateom actually cancels the request if it times out.

Yes, we should make sure the actor networking stays up during this hook. Good point on the additional documentation with the mismatched timeouts.

Would it be possible to send traceparent with the hook requests? Then whatever the actor does in the hook shows up in the same trace as the suspend or resume. It would make slow resumes a lot easier to debug.

I'd have to look into this, but there is likely other information we want to pass through to the hook (or if not through the hook directly, through another API that the actor can query). Regardless we should provide a way for actors to get the traceparent during the hook.

Fail open sounds right, but I'm a bit worried failures will be pretty hard to notice when logged only. What do you think about a hook duration metric and a failure event with the actor's identity? I'm happy to track these in new issues.

Sure, we can have additional metrics around this.

Btw, I'm assuming hooks don't fire on delete, so the final flush there still relies on SIGTERM?

Yes that's correct, at least for now. There is an idea to auto-suspend on termination instead of sending SIGTERM #1707.

Navigation Menu

Repository navigation

Actor lifecycle hooks #450

Actor lifecycle hooks: `preCheckpoint` and `postRestore`

Problem

Design

API

`preCheckpoint`

`postRestore`

Behavior

Integrating

Appendix

Why GET, not POST

Why not fail-closed

Waiting for the current turn

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Actor lifecycle hooks #450

Description

Activity

EItanya commented on Jul 20, 2026

eliranw commented on Jul 21, 2026

HavenXia commented on Jul 24, 2026

sfunkenhauser commented on Jul 27, 2026

HavenXia commented on Jul 27, 2026

sfunkenhauser commented on Jul 27, 2026

sfunkenhauser commented on Sep 24, 2026

Actor lifecycle hooks: preCheckpoint and postRestore

Problem

Design

API

preCheckpoint

postRestore

Behavior

Integrating

Appendix

Why GET, not POST

Why not fail-closed

Waiting for the current turn

krisztianfekete commented on Sep 24, 2026

sfunkenhauser commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Actor lifecycle hooks: `preCheckpoint` and `postRestore`

`preCheckpoint`

`postRestore`