Repository navigation
[Feature] External volume mounts #232
Description
Continuing the discussion in #119 (comment) and #220, let's start discussing the use cases and requirements for external volume mounts.
Use Cases
One use case I see discussed in the above issues is the need for each actor to have their own workspace to store data. Rootfs or working dir is insufficient because:
- At larger capacities, there is a performance penalty of checkpointing local directories every time an actor is paused/suspended
- Actor requires higher persistence guarantees with every IO rather than at the pause/suspend timeframe.
Open questions
- Do volumes need to be sharable across multiple actors? If so, as read-only or writeable?
Proposal
Here's my initial thoughts on external volume semantics we can add to substrate.
Lifecycle
External volumes are an independent filesystem mount that follows the Actor lifecycle. A unique volume will be created per actor. Once a volume is created, it is persisted across the lifecycle of the actor. It is deleted when the actor is deleted.
If we support actor updates in the future, the volume will persist across updates. A volume can be added/removed across updates, but an existing volume cannot be mutated (for now). When a volume is deleted from the template, the underlying volume is also deleted after the rollout has completed.
Data Persistence Semantics
External volume mounts are expected to provide filesystem consistency semantics. Writes are persisted when flushed to the backing store. This is different than rootfs semantics where data is only guaranteed to be persisted when the actor is paused/suspended.
When an actor is paused/suspended, we do not take a snapshot of the volume mount (at least in the initial phase). We just unmount the volume and remount it when the actor is resumed. This implies that on resumption, the actor will get the latest volume state, regardless of if the actor gracefully completed its previous task or it crashed prematurely. The actor has to be able to handle the potential difference between its memory state and filesystem state.
In a future milestone, we can explore adding external volume snapshot support on actor suspension, and the ability to create actor volumes from a given volume snapshot.
Storage Backends
The storage backend has to support the following functionality at the required performance / scale:
- Create/delete volumes
- Attach/detach volumes to nodes
- Mount/unmount volumes to nodes
- Security/isolation of volume mounts
- (Future) snapshot of volumes and clone from snapshot
NFS appears to be a commonly requested storage integration for this use case, however, while the file protocol is standard, the control plane APIs are typically vendor-specific. Rather than add a hard-dependency on NFS in substrate, I propose that we focus on supporting Container Storage Interface (CSI), which is well-defined and supported in the Kubernetes ecosystem.
API
kind: ActorTemplate
metadata:
name: my-agent-template
spec:
...
containers:
- image: my-agent-image:v1.2.3
volumeMounts:
- name: vol1
mountPath: /my-data
volumes:
- name: vol1
externalVolumeTemplate:
capacity: 5Gi
storageClassName: storage-pool1
Activity
msau42 commentedon Jun 12, 2026
chenggui53 (@chenggui53) would love to hear more about your use case and if this proposal could address it.
chenggui53 commentedon Jun 15, 2026
Michelle Au (@msau42) Thanks for your support on this problem.
In our primary scenario, we need to maintain many agent instances (e.g., OpenClaw/Hermes) within a substrate actor. Each agent instance must persist its own data when the instance is paused, or when its node crashes – basically any fault that could cause in-memory data loss.
Therefore, I think a persistent volume with the same lifecycle as the actor could solve part of our problem.
As an open question: sometimes we also need a volume that can be shared among actors within a certain scope (e.g., a namespace). Such a shared volume could hold read‑only data maintained by an admin.
So I’d like to ask: is it possible to provide both actor‑lifecycle volumes and shared volume mounts for actors?
EItanya commentedon Jun 15, 2026
I definitely think we need both of these.
Shareable, definitely. Writable gets potentially difficult 😆. I think it could offer some powerful sharing potential, but would need to be used with caution.
chenggui53 commentedon Jun 16, 2026
Maybe use it just like pvc for a pod.
MushuEE commentedon Jul 8, 2026
+1 to shared. CUJ: I am running GasTown with multiple "Pole Cats" all operating on the same shared context they write and share scripts, findings, etc to a mounted volume. Managing an independent volume lifecycle will get complex I imagine and if Actors are scheduled in a way that lands them into the same node as other Actors, cross-region etc. I am unsure if they will be able to share volumes across networks.
msau42 commentedon Jul 24, 2026
Initial PR to add per-actor volume API and lifecycle hooks is here: #405.
It's only useable in a test/dev purpose right now, we're actively working on adding CSI support in parallel.
Here are the followup items from the initial PR reviews for tracking:
Handled by #527:
Handled by #678
Discussing as part of #643:
Open items:
Add basic per-actor external volume flow (#405)
Add validation to require that every volume is specified in volumeMou…
Handle partial volume creation failures (#527)
mowangdk commentedon Aug 18, 2026
Hey team — I'm the maintainer of Alibaba Cloud's CSI driver. I have few concern that ties this whole approach together, and I'd love to hear how you're thinking about it.
Routing every CSI volume through guest → virtio-fs → vhost-user → virtiofsd → host bind-mount → backend adds a FUSE/vhost-user hop plus cache-revalidation on each IO — versus just handing the guest a block device.
So the perf-optimal direction seems to point away from virtio-fs.
Running a single instance of the CSI plugin/virtiofsd per node exposes all tenant mounts in a shared tree under one worker identity.
Since all volumes share the same host FS namespace, isolation relies solely on directory paths and file permissions. A compromised worker or an actor with elevated privileges could traverse the tree to access other tenants' volumes, violating multi-tenant isolation guarantees.
Questions
Enable CSI support for micro-vm (#840)
BenTheElder commentedon Aug 20, 2026
Move fast. Also, FWIW virtio-blk passthrough seemed to have serious issues in developer environments (limactl / mac / kind), which is a key concern for us, but not more important than performance.
That comment is stale, it didn't work well for that purpose, provisioning the block devices is slow and it wasn't really any more reliable. Worth revisiting.
There's a virtiofs per actor, not per node.
We are only running hardened sandboxes like gVisor and uVM, an escape of those is pretty bad for multiple reasons currently, I don't think this is top of the list.
BenTheElder commentedon Aug 20, 2026
In general I would describe the uVM implementation as "proof of concept". We started with gVisor, it's a lot more mature at the moment.
mowangdk commentedon Aug 21, 2026
Gotcha, Thanks for following up~
identity: Secure CSI Driver TCP Connections via Pod Identity (#956)
docs: add guide for CSI based external volumes (#1072)