Repository navigation
Need to consider snapshot locality during assignment, and do local caching #52
Description
NIC bandwidth is finite. As we scale up the number of turnovers per second, we will exhaust that bandwidth with snapshots.
We will need to manage NIC usage explicitly. When we take a snapshot, we might have to keep it locally for a while. We need to do something like:
- Publish that we have snapshot X on this node
- When a request comes in for actor A which needs snapshot X:
- route it to a worker on a node which has snapshot X if possible
- If not possible, see if the data exists in the backing store (meaning any node can restore)
- If so pick any worker
- If not, park the request and retry
Depending on how fancy we need to get we might notify the storage on the node that does have the snapshot that it should move that upload to the front of the queue.
Activity
Cali0707 commentedon May 26, 2026
Tim Hockin (@thockin) I think one approach to tackling this is (divided into separate PRs):
ResumeActorflows check the node cache (e.g./run/ateom-gvisor/snapshot-cache) before falling back to pulling from the network. Snapshots are only eligible for eviction once they've been uploadedDoes this approach sound reasonable to you? If so, I'm happy to proceed with PRs
thockin commentedon May 31, 2026
I think you're on the right track. Let's merge this discussion with #119 which has a lot more details that are related. Also #44