Repository navigation
Description
Problem
Every successful suspend creates an immutable ActorSnapshot, but superseded and deleted-Actor snapshots are never removed. An overnight soak produced 1,603 generations for one Actor, of which 1,602 were no longer referenced. Bucket lifecycle rules are unsafe because they cannot distinguish an old live snapshot from garbage.
Retention semantics
A snapshot is protected when any of these is true:
- an
Actor.latest_snapshotreferences it; - an
ActorSnapshotTagreferences it; - it is among the newest
minimumCountsnapshots produced by its live source Actor UID; or - it is younger than
minimumAgeSecondsand its source Actor still exists.
Add user-tunable policy to ActorTemplate.spec.snapshotsConfig:
retentionPolicy:
minimumCount: 3
minimumAgeSeconds: 86400Both thresholds use OR semantics. minimumCount includes the Actor's latest snapshot. Explicit zero disables that threshold; omitted fields default to 3 and 86400. Policy updates apply to existing history on the next GC pass. Actor UID, rather than name, defines history so delete/recreate cannot inherit it.
Deleting an Actor removes count/age protection from its history immediately. Tags or references from other Actors still pin those snapshots.
Collection design
- Run a periodic collector in
ateapi. Use Kubernetes Lease leader election so only one replica collects at a time; the existing Redis distributed lock remains an alternative if Lease coordination proves unsuitable. - Determine retention from authoritative Actor, tag, snapshot, and ActorTemplate state. Before deletion, prevent new references and recheck roots so tagging/cloning cannot race GC.
- Delegate deletion of the exact private snapshot prefix to any healthy
atelet;ateletalready owns the GCS/S3 clients and credentials. Add an internal idempotent delete-prefix RPC and backend implementations. - Delete object-storage bytes first, then snapshot metadata. On any failure, retain metadata and retry on a later pass. Never delete metadata first, which would make leaked bytes undiscoverable.
- Keep GC interval/concurrency as operator settings; retention remains workload policy.
- Emit basic attempted/deleted/failed counters and structured failure logs.
Out of scope
- Partial failed uploads with no
ActorSnapshotmetadata (#610). - Node-local Actor directories and pause snapshots (#641).
- Per-Atespace policies or storage quotas.
- Manual
DeleteActorSnapshotAPI.
Acceptance criteria
- CRD defaults retain the newest 3 snapshots and all snapshots younger than 24 hours for each live Actor.
- Users can tune either threshold, including setting either or both to zero.
- Latest-snapshot and tag references are never collected.
- Deleting an Actor makes its otherwise-unreferenced history collectable.
- Concurrent tag/clone operations cannot produce a reference to deleted data.
- GCS and S3-compatible prefixes are deleted idempotently; metadata is removed only afterward.
- Kubernetes Lease leader election makes collection safe with multiple
ateapireplicas, and collection resumes after process/storage failures. - Unit tests cover policy boundaries, references, races, and retry ordering; E2E verifies superseded and deleted-Actor cleanup while tagged snapshots survive.
Activity
thockin commentedon Aug 17, 2026
A few thoughts.
Snapshots and tags are similar to
gitcommits, tags, and heads. What can we learn from git's GC policies? Those are pretty well tested.Think of actors as git heads. Each actor is a branch. It has a "current" commit (snapshot). Each commit (snapshot) has a parent commit (snapshot), but git doesn't have to retain all of the history (
--depth=N).Our problem is a little different - git considers the commit parentage to be protected, and we are specifically saying we want to GC those. So it's more like a "floating" depth.
Git's rules:
That's it (as far as I can tell). If you add in the depth rule it is:
Can we think really hard about the policies we need to express?
I think
minimumCount== RETENTION_DEPTH in my notes?I do not understand
minimumAgeSeconds? Is it meant to be RETENTION_TIME? The word "minimum" doesn't express that to me.Do tags count against minimumCount? That seems wrong. I see the need for a max N, but I think I would express it as:
Thoughts?
igooch commentedon Aug 17, 2026
A failure case the design doesn't cover: a partial listing during collection
Retention here is proved by absence — a snapshot is collectable because no tag references it, and because its source Actor is gone or superseded. So anything the collector fails to read is indistinguishable from something that doesn't exist:
The design covers deletion failures (retain metadata, retry) and reference races (recheck before delete), but both assume the listings came back complete.
Where we hit this
Same shape in the image-cache GC (#463 Phase 2), found during code review of the eviction engine: an unreadable image record contributed no refcounts, so the layers it referenced looked unreferenced — one corrupt file converting into deletion of other, healthy data.
Why we fixed it by gating the whole pass (not skipping the unreadable item)
The two failure sides aren't symmetric:
Mechanically: any scan that can't complete returns a sentinel (
ErrIncompleteEnumeration), the pass deletes nothing, logs once naming the unreadable path, and retries next pass.Does it apply here?
Depends on one property: are the store's listings all-or-nothing?
Sneha-at commentedon Aug 19, 2026
I have started looking into the snapshot cleanup. I want to confirm some assumptions before proceeding.
A) Are we onboard on building up on ActorSnapshot resource?
B) Assuming the snapshot information is stored in both Postgres DB and remote storage (S3/GCS) and thus would have to be cleaned up from these places (excludes local storage cleanup)
C) Right now we want to focus on the snapshot storage cleanup and the signed URL access would be implemented in a later release.
Cleanup design key points -
laoj2 commentedon Aug 26, 2026
Edit: this design was superseded by #664 (comment)
I'm taking this over from Sneha-at (@Sneha-at). Below is a summary of the proposed plan for remote snapshot garbage collection,
I'd love to get your thoughts and feedback before moving into implementation.
Note: I'm focusing on GC for external/remote snapshots for now. Local snapshots are being garbage collected in the common case(see
substrate/cmd/atelet/local_checkpoints.go
Line 32 in 24b997c
Scope & Context
This design focuses strictly on external/remote snapshots stored in object storage (GCS/S3).
Local snapshots are already pruned on-node for the common cases (see local_checkpoints.go#L32, triggered during
CheckpointandUploadPausedCheckpoint). We still need a follow-up audit to verify all local edge cases are covered, but that is orthogonal to remote GC.External snapshots currently leak: every checkpoint uploaded to object storage remains there indefinitely once superseded or once the actor is deleted.
Non-goals for v1
To keep the initial implementation focused and avoid scope creep:
retentionPolicy: no time-based TTLs or "keep last N snapshots" settings.Retention rule:
A remote snapshot is considered live if and only if it is referenced by either:
status.latest_snapshot).snapshottarget).Everything else is garbage and eligible for collection.
Note that golden snapshots meet the retention criteria above, because it's pointed to by the golden actor. In the future, we should make this more robust.
Storage Layer: FK-enforced retention
latest_snapshot_atespace,latest_snapshot_nametoactorstable with aFOREIGN KEY … ON DELETE RESTRICTtoactor_snapshotsDELETE FROM actor_snapshotsthen fails with a FK violation if the snapshot is tagged or is still some Actor's latest snapshotactor_snapshot_deletionsas an async deletion queue: that will hold the to-be-deleted snapshot urisWhy do we need to project snapshot_atespace/name in
actorstable?A snapshot is not private to one Actor. N actors cloned from one tag all point at the same row until each takes its own first suspend.
hat makes the snapshot garbage collection a cross-actor operation, and the tag is not a sufficient guard -> because tags can be deleted via DeleteActorSnapshotTag RPC:
So, imagine the following scenario:
Answering is
Sreferenced by any other actor would require scanningActorstable and the snapshot details are opaque in the actor proto.Workflow
SuspendActor,DeleteActorandDeleteActorSnapshotTagoperations: push candidates to a deletion queue.* Snapshot retention decision is delegated to the storage layer (this is enforced by foreign keys)
SuspendActor Workflow
DeleteActor Workflow
DeleteActorSnapshotTag workflow
Background periodic collector
A periodic worker in ateapi processes queued deletions asynchronously:
Open Questions
Should ateapi delete the snapshots directly, instead of delegating to atelet?
Do we need a GC that scans the whole snapshot table in the future? The proposed GC is event-driven, so it doesn't self heal:
* In the proposed workflows, mutating the primary resource and running
CollectActorSnapshotare separate transactions, so if ateapi crashes after the actor was deleted/suspended but before we callCollectActorSnapshot, we would leak that snapshot forever.laoj2 commentedon Aug 27, 2026
I also audited where we are be leaking local snapshots:
atelet is the only thing that can delete node-local snapshots today, and the
control plane reaches it only through
Status.WorkerAssignment(e.g.substrate/cmd/ateapi/internal/controlapi/workflow_delete.go
Lines 116 to 120 in 75c9b14
Terminate()ran but didn't prune snapshots. Fixed by Delete local snapshots when an actor is terminated #1246.WorkerAssignmentfirst.ActorCrashedMetadata, so the actor never crashes and never leaves SUSPENDING. It keeps the snapshot, and deleting the actor hits the nil-assignment skip in the control plane.
moves on. So a failure on Delete operations is permanent
NodeVmsWithLocalSnapshotsholds at most one entry, but the fix needs to be extendible.Delete local snapshots when an actor is terminated (#1246)
laoj2 commentedon Sep 1, 2026
The proposal in #664 (comment) introduced yet another FK, which we've been trying to avoid, given the scale we want to support.
So, we took a step back and rethought how snapshots and tags should be stored/represented in the system in the first milestone. This is the updated proposal for modelling snapshots/tags in the API and how to garbage collect them:
cc Eitan Yarmush (@EItanya) Julian Gutierrez Oschmann (@juli4n) Dmitry Berkovich (@dberkov)
TL;DR
We propose a snapshot ownership model that is single and static:
suspend.Garbage Collection: whoever created/owns the snapshot is the only one who ever deletes them.
This proposal:
actor_snapshotsactor_snapshot_tagsat the size it already is onmain, and keepsactor_snapshot_tags_atespace_fkGetActorSnapshotand turnsListActorSnapshotsintoListActorSnapshotTags: tags become the only named, listable handle on a snapshot. Keeps the 4 tag RPCsCurrent State
DB Model
Snapshots are referenced by 3 tables:
Ownership Model
None. Actor, Tag and Golden Actor all reference a snapshot and none of them owns it today.
Snapshot Lifecycle
Birth
SuspendActorstores the snapshot in GCS/S3 and in the DBTagging
ATESPACE, readable only inside its own atespace, orPUBLISHED, clonable from anywhereCloning
CreateActorpointsactor.status.latest_snapshotat an existing tagged snapshotReading
actor.status.latest_snapshotinto the snapshot row and pullssnapshot_uriandcontent_scopeoff itGC Eligibility
Snapshot is not referenced by any tags or actors → requires search across multiple tables (including opaque fields)
Proposal
DB Model
actor_snapshotsis dropped. The fields that lived in its proto move onto the Actor and onto the tag tables, as an inlineExternalSnapshotmessage (snapshot_uri+content_scope) on whoever holds the snapshot.actor_snapshot_tagsloses its FK intoactor_snapshots, its index, and its two snapshot columnsstatus.current_snapshot_tagis a pointer inside the Actor's proto, not an FK: nothing in the schema stops the tag from being deleted first. The only FK left isactor_snapshot_tags_atespace_fk, which is what keeps an Atespace from being deleted out from under its tags.Ownership Model
Ownership is single and static: the owner is fixed when the objects are written, and never moves. A tag does not take a snapshot over from an Actor, it gets its own copy.
actor.status.external_snapshotSuspendActor, and atDeleteActortag.status.snapshotCreateActorSnapshotTagcopies the Actor's snapshotDeleteActorSnapshotTagCreateActorfrom a tag; no objects are writtenstatus.external_snapshotSnapshot Lifecycle
Birth
SuspendActorstores the snapshot in GCS/S3 and writes its URI onto the Actor rowactor_snapshotstablestatus.current_snapshot_tagis cleared in the same commit that records the new snapshotTagging
CreateActorSnapshotTag(T, actor=A)is kept, and names the Actor to tag rather than a snapshot resource that no longer existsstatus.in_progress_snapshot_uri, before a single object is writtenstatus.snapshotand clear the in-progress fieldSUSPENDEDand hold a snapshot: a running or suspending Actor's is either stale or still being writtentag-prefixed UUID, recorded on the row and only ever read back. Nothing can compute its way onto another tag's objects, and a recreated tag never inherits what its predecessor strandedstatus.source_actor_uid: adopting another Actor's pending create would interleave two sources into one prefixCreateActorrefuses to seed from one andUpdateActorSnapshotTagrefuses to change its scope, since both would hand out content that is still being written or may never beATESPACEorPUBLISHED, and can be changed afterwards viaUpdateActorSnapshotTagCloning
CreateActorcopies the tag'sExternalSnapshotintoactor.status.external_snapshotand records the tag inactor.status.current_snapshot_tagGC Eligibility
Three release points, all synchronous.
SuspendActorcurrent_snapshot_tagwas unsetDeleteActorcurrent_snapshot_tagis unsetDeleteActorSnapshotTagEvery release runs before the row that names the objects is overwritten or dropped, so an interrupted release is rediscoverable and the retry deletes whatever is left; the opposite order would drop the only handle on them. A failure fails the whole RPC and the client retries with the same arguments. The cost is that a crash between the release and the commit can leave objects no row names — strictly better than losing the ability to ever delete them.
One case still leaks, and is marked TODO: an in-progress snapshot whose ActorTemplate no longer resolves. The template's
storageLocationis the only place its URI can be derived from, so without it the choice is between leaking the objects and wedging the Actor inDELETINGforever.Open Questions / Future Work
CreateActorSnapshotTag())CreateActorSnapshotTag(actor=A)stays, andSuspendActorgains no tag parameter. Tagging is a copy that can take a while, and folding it into suspend would hold the actor's lease for the length of itCreateActorracing aDeleteActorSnapshotTagcan seed from content that is going away: the tag stays resolvable while its snapshot is being collectedGarbage Collect snapshots and remove the snapshot resource (#1417)
Call the generated validation fns for ActorSnapshotTag methods (#1488)
Rename ActorSnapshotTag to Tag (#1491)
Move DeleteTag orchestration to a workflow (agent-substrate#1720)