Repository navigation
keep a microVM container's rootfs upper on a virtio-blk ext4 disk #3153
Description
On the microVM runtime with EROFS lower layers (#2440), the guest mounts read-only image layers over virtio-pmem with DAX, while the container's writable overlay upperdir and workdir still live on the virtio-fs share (<cid>/rw/fs and <cid>/rw/work) served by virtiofsd.
Keeping the writable upper on virtio-fs costs us in two places:
- Every guest write, create, rename,
stat, or unlink in the upper is a FUSE round-trip throughvirtiofsdand a host VFS syscall on the node's ext4 filesystem. Agent workloads (git checkout,pip install, builds, test runs) are dominated by small-file metadata operations, and under concurrent snapshot writebackvirtiofsdblocks on the same host ext4 journal asatelet. - Checkpointing walks
fs/andwork/file by file to buildrootfs-rw-<cid>.tar, and restore untars every file back onto the host filesystem. The cost scales with file count rather than byte size.
Proposal
Give each EROFS microVM container its own sparse ext4 disk image (<cid>.upper.ext4) attached over virtio-blk:
- Cold boot the seeded (
resolv.conf,hosts,hostname) upper into a 4 GiB sparse journal-less ext4 image, attaches it read-write as/dev/vd{a,b,...}; Kata agent mount it withnoinit_itableat/run/kata-containers/upper/<cid>. The container overlay takesupperdirandworkdirfrom there. - Checkpoint hard-links
<cid>.upper.ext4into the checkpoint directory asrootfs-disk-<cid>.ext4in place of therootfs-rwtar. In aMEMORYsnapshot, the guest's dirty ext4 pages are captured inmemory-ranges. For aROOTFSsnapshot, the worker removes the container and destroys the sandbox before archiving so the guest unmounts and flushes the disk cleanly. - Restore copies
rootfs-disk-<cid>.ext4out of the restore directory withcopy_file_rangeand rewrites the disk path in Cloud Hypervisor'sconfig.json.
Benchmarks
- Below PoC & micro-benchmarks are handled in-guest
fioand POSIX metadata on/workat overlay upper - 2 vCPU / 2 GiB microVM actor on an
c3-highcpu-192-metalnode - medians across 6
virtio-fsruns and 3virtio-blkruns):
Workload on /work (overlay upper) |
virtio-fs upper | virtio-blk upper |
|---|---|---|
| Sequential write (1 GiB, 1 MiB bs, psync, end_fsync=1) |
734 MiB/s (1,394 ms) | 1,108 MiB/s (923 ms) |
| 4 KiB random write (1 GiB, O_DIRECT, libaio qd=16) |
17.8k IOPS (p50 819 us, p99 1,011 us) | 73.2k IOPS (p50 73 us, p99 148 us) |
| 4 KiB random read (1 GiB, O_DIRECT, libaio qd=16) |
101.4k IOPS (p50 76 us, p99 444 us) | 196.3k IOPS (p50 50 us, p99 89 us) |
fio smallfiles (10,000 x 4 KiB create + write + close) |
3,178 ms (3.15k IOPS) | 101 ms (98.0k IOPS) |
| Metadata: create 20,000 empty files | 6,405 ms | 214 ms |
Metadata: ls -l over 20,000 files |
664 ms | 54 ms |
Metadata: rm -rf 20,000 files + sync |
2,456 ms | 286 ms |
SWE-perf (swebench-astropy-7336, 150 users for 10 min on c3-highcpu-192-metal, 3 worker pods, MEMORY snapshots to GCS):
| 150 users, 10 min (24 MB upper) | virtio-fs upper | virtio-blk upper |
|---|---|---|
| tasks done / suspends / failures | 474 / 2,140 / 0 | 580 / 2,552 / 0 (+22% tasks) |
| ResumeActor p50 / p99 (s) | 1.4 / 39.0 | 1.4 / 4.4 |
| SuspendActor p50 / p99 (s) | 2.1 / 88.0 | 2.7 / 97.0 |
| TaskCEL p50 / p99 (s) | 7.5 / 44.0 | 6.7 / 8.1 |
| TaskWallClock p50 / p99 (s) | 61.0 / 288.0 | 47.0 / 178.0 |
| worker checkpoint avg (s) | 0.41 | 0.31 |
| worker restore avg / sampled p99 (s) | 0.52 / 6.37 | 0.28 / 0.66 |
restore upper step (upper_join) p50 / p99 (s) |
0.03 / 5.71 (untar) |
0.02 / 0.05 (copy_file_range) |
| snapshot upper avg (MB) | 24.2 (tar) |
24.8 (ext4 image) |
| node IO PSI full avg | 39.4% | 32.6% |
Results
- In-guest small-file writes and metadata ops (
create,stat,unlink) are 9x to 31x faster since they stay in the guest ext4 filesystem instead of doing a FUSE round-trip per syscall. - On
OnDemandmemory restore, Cloud Hypervisormmapsmemory-rangesin ~0.2 s, so untarring a 24 MB upper (5.71 sp99) is 90% of the worker restore tail (6.37 sp99). Copying one sparse image takes0.05 sp99 (0.66 sworker restore p99) and cutsvirtiofsdand workerD-state stalls by >95% at 150 users.
Trade-offs to discuss
- Disk and inode limits: the 4 GiB sparse image is formatted with 65,536 inodes (~150 KiB initial size), so containers exceeding 4 GiB or 65k files hit
ENOSPC. We can raise the default virtual size / inode count (unallocated inode tables are lazy and cost very little on disk) or size it from the actor template. - Discard on delete: deleted files inside the guest don't free blocks in the host sparse file unless we mount the upper disk with
discard(so guest deletes punch holes on the host). - Packaging: needs a static
mke2fs(e2fsprogs 1.47+ for-d) in the worker image orSandboxConfigasset
Related:
#2440 serve container image lower layers as EROFS to microVM (DAX) and gVisor
#1954 / #3026 drop per-file fsync when tarring rootfs-upper
kind/design
area/microVM
area/storage
Activity
ajaysundark commentedon Oct 23, 2026
cc Michelle Au (@msau42) Benjamin Elder (@BenTheElder) Walter Fender (@cheftako)
ajaysundark commentedon Oct 23, 2026
Other general virtio-fs vs virtio-blk perf discussions:
This also aligns with our interest on giving dedicated block-based snapshot storage for actors -- when an actor has its own attached block disk, the ext4 upper image can simply live directly on that disk. Together, they completely eliminate the need to tar files inside the VM and eliminate the need to copy the resulting snapshot across the network to GCS.
BenTheElder commentedon Oct 23, 2026
We need to take a hard look at avoiding that.
Also, not entirely sure ext4 is what we want, and we have to consider some other metrics missing here, like small durdir lower bound size.
However I do think we want to move away from virtiofsd for actors without CSI volumes, post-GA.
That does not follow, nothing about this eliminates network bandwidth. If anything the opposite, tar is smaller.
There are some other useful properties to block though, unlike tar we can let the object store plugin write this directly with a parallel download while having no subsequent decode step to coordinate.
BenTheElder commentedon Oct 23, 2026
Also there's a significant trade-off in that we have to set an upper bound, and that influences the metadata overhead IIRC. Tar metadata scales with what the user uses and doesn't really have an upper bound. We're still getting signal on how users will use durabledirs. 4GB may be too small.
Previously I discarded this for those reasons. I do think we MUST revisit it, but there are some dimensions missing in the op.
ajaysundark commentedon Oct 23, 2026
Agreed on avoiding additional assets. The guest already ships mkfs.ext4 today, I had to use mke2fs to seed the /etc/resolv.conf, hosts, hostname files etc. We may be able to handle this directly in the worker.
On the fs choice, I was experimenting few options. journal-less ext4 was great for small lower-bound cases (empty 4GB sparse image takes ~150KB of allocated blocks). I also looked at xfs, which is better performing, but enforces some minimum size restrictions. We need to weigh the tradeoffs.
This issue / proposal is primarily for evaluating blk-storage for rootfs-upper first.
Fair point, that was overstated in the description. I could reword it better. I was thinking along #1951 as local NVMe caching where the block image could stay on a volume / snapshot backend.
agreed, I also called out the cap limitations, we need to look into it further. 4GB was arbitrary. My agent PoC with 'lazy_itable_init' left unallocated block groups/sparse, so even if virtual cap as a larger 32 or 64 GiB wouldnt cost us as much. We maybe able to size the cap w. pod ephemeral storage limits?
That makes sense. I've also been experimenting with CoW backing for the virtio-blk uppers, for eg: thin-pool with CoW and XFS/reflink. The results are very convincing for golden-clones and restore. Happy to add the results to the issue to open up further discussion / evaluate overall fit for rootfs upper
BenTheElder commentedon Oct 24, 2026
I dug up my old prototypes with ext4, and confirmed that I'd already worked around that and we can re-use that part if we go with ext4.
Basically I took an ext4-in-go library designed to create read-only layers for hcsshim and forked it to support rw (not really in scope for upstream). It's not totally terrible, though I wouldn't necessarily go that route off the cuff ... it's a fair bit of code, and still missing features, does perform kinda decently and avoids the packaging issues:
main...BenTheElder:ext4-writer
There's an alternative the possibility of pre-provisioning a "golden ext4" as a cached sandbox asset and just patching the UUID which might be more elegant (and faster, if we're using fixed size sparse disks anyhow)
main...BenTheElder:substrate:ext4-golden-uuid
I think I might like that more ... it's not much code to patch the UUID (so each disk can be unique against the pre-provisioned golden) and still avoids packaging issues. However, only if we selected a fixed size (or maybe a small number of fixed sizes).
Yeah, they have some .... very interesting tradeoffs. We probably need to tag in some more folks to consider as we get closer.
.. maybe, though then the cap depends on the worker, in a way behavior generally shouldn't depend on the worker ... (sure the worker and the node ultimately do have actual limits, but the same actor template landing on different workers getting different disk sizes is ... surprising)
Yeah, this also depends on some work Michelle Au (@msau42) is prototyping that changes a few of the dimensions, like ... we might not really care about the cost to write any form of durabledir in some environments in the future.