KVM: createVolume / createTemplate from incremental snapshot fails randomly, managesnapshot.sh skips mkdir #14485

problem

With kvm.incremental.snapshot=true and snapshot.backup.to.secondary=false, createVolume or createTemplate from an incremental snapshot fails randomly with:

errorcode 530
Failed to copy snapshot: java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback

The agent log only says Failed to backup <uuid> ... to <secondary>/snapshots/<account>/<volume>. The real error is hidden because managesnapshot.sh sends qemu-img's output to /dev/null. I traced it on the host:

qemu-img convert ... : Could not create '<secondary>/snapshots/2/3/<uuid>': No such file or directory

What happens:

1. Both calls copy the snapshot into snapshots/<account>/<volume> on secondary. After the call, the SSVM deletes the copy and also removes the folder if it's empty (NfsSecondaryStorageResource.java#L2116-L2120).
2. On the next copy, backup_snapshot() only creates the folder if [ ! -d $destPath ] (managesnapshot.sh#L237-L245).
3. If the host has the same secondary export mounted somewhere else (configdrive, template), the NFS client can still see the folder the SSVM just removed. mkdir is skipped and qemu-img fails (managesnapshot.sh#L285).

Every failed copy I traced had no mkdir before qemu-img. Every good one had it.

host-trace.log

versions

• CloudStack 4.22.1.1, same code on main (0a5bf30)
• KVM: Ubuntu 26.04, QEMU 10.2.1, libvirt 12.0.0
• NFS 4.2 primary and secondary (Linux NFS server)

The steps to reproduce the bug

1. KVM zone with NFS secondary. Set kvm.incremental.snapshot=true and snapshot.backup.to.secondary=false, restart the management server.
2. On a KVM host, keep a subfolder of the same secondary export mounted, like a configdrive mount does:
   mount -t nfs -o vers=4.2 <secstore>:/<export>/configdrive /mnt/keep
3. Deploy a VM on that host and take a full snapshot of its ROOT volume.
4. Loop: createSnapshot (incremental), then createVolume from it, then createTemplate from it.

Without step 2 I got 0 failures in 8 rounds. With it, 6 of 12 rounds failed, all with the error above.

api-results.txt

mgmt-failed-jobs.log

Same failure shows in the UI ("Failed to create volume"):

What to do about it?

• Run mkdir -p "$destPath" every time in backup_snapshot(). It's safe if the folder exists.
• Keep qemu-img's error in the message sent back, so this shows up in the logs.
• Maybe don't remove snapshots/<account>/<volume> when the temp copy is deleted.