Repository navigation
KVM + Linstor: detachVolume reports success but leaves the disk attached when the domain XML still has the legacy /dev/drbdNNNN path #14508
Description
problem
Since #12300 (4.20.3.0), the Linstor adaptor returns /dev/drbd/by-res/<rsc>/0 as the device path of a volume. If a data disk was attached before the upgrade, and its VM has not been stopped/started via CloudStack since, the live domain XML still has the raw DRBD minor path, e.g. /dev/drbd1069. This is the same legacy-path situation that #13538 fixed for NAS backups.
On detach, attachOrDetachDisk(attach=false) looks the disk up in the domain XML by an exact path comparison:
/dev/drbd/by-res/cs-<uuid>/0 does not equal /dev/drbd1069, so the lookup fails and the agent logs
Could not find disk [/dev/drbd/by-res/cs-<uuid>/0] attached to VM instance with UUID [...]. We will set it as detached in the database to ensure consistency.
The agent then returns without sending detach-device. dettachVolume calls disconnectPhysicalDisk and answers success. The volume becomes Ready with no VM in the DB. However, the disk is still in the domain, QEMU keeps the DRBD device open, and the guest never gets an unplug request.
What we saw after such a "successful" detach (VM-A holds the stale disk):
- Attach to a VM on another host fails:
org.libvirt.LibvirtException: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/dev/drbd/by-res/cs-<uuid>/0': Wrong medium type - Attach back to VM-A fails:
org.libvirt.LibvirtException: XML error: target 'vdb' duplicated for disk sources '/dev/drbd1069' and '/dev/drbd/by-res/cs-<uuid>/0' - Attach to another VM, apparently on the same host as VM-A, succeeds. The same DRBD device is then attached to two running VMs at once. We had two volumes in this state. The stale copies were not mounted, but nothing prevents both guests from writing to the device.
Guest-side workaround: in the VM that still has the disk, run echo 0 > /sys/bus/pci/slots/<N>/power. The guest ejects the device within a second, libvirt drops it from the domain, and the volume can be attached elsewhere. This also shows that ACPI hotplug works in the guest: it just never received a request.
Note: we are a tenant and have no access to the agent logs. The log line above is from reading the code. The rest (API events and results, libvirt errors returned by attachVolume, guest state) we observed directly.
versions
- CloudStack 4.20.3.x, KVM, Linstor primary storage. The volume was attached before the cloud was upgraded from a release without linstor: Provide /dev/drbd/by-res/ resource paths to CloudStack #12300.
- QEMU / libvirt / DRBD / LINSTOR versions: unknown to us.
- Guests: Ubuntu 24.04, kernel 6.8, PCI (ACPI) hotplug. Volumes are attached/detached by the leaseweb cloudstack-csi-driver 0.10.0, which only calls
attachVolume/detachVolume.
The steps to reproduce the bug
- On a release without linstor: Provide /dev/drbd/by-res/ resource paths to CloudStack #12300 (e.g. 4.20.2.0) with Linstor primary storage, attach a data volume to a running VM.
- Upgrade management and KVM agents to 4.20.3.x. Do not stop/start the VM.
detachVolumethe volume: the job succeeds and the volume has no VM.- On the host,
virsh domblklist <vm>still shows the disk as/dev/drbdNNNN, and the guest still sees it. attachVolumeto a VM on another host fails withWrong medium type. To the same VM it fails withtarget ... duplicated. To a VM on the same host it succeeds.
What to do about it?
- When looking up the disk to detach, compare resolved paths (
/dev/drbd/by-res/<rsc>/0is a udev symlink to/dev/drbdNNNN) or match by disk serial (diskUuidToSerial(vol.getUuid())), not by the raw source string. - Do not report a successful detach when no disk matched the path but a disk with the volume's serial is still in the domain. That is the case where "set it as detached in the database to ensure consistency" is unsafe.
- Optionally, refuse an attach while the Linstor resource is still in use by another domain, outside of live migration.
Related: #12300 (path change), #13538 (same legacy path, fixed for NAS backup), #14214 (orphaned disk in libvirt after detach, different cause).
Activity
boring-cyborg commentedon Oct 28, 2026
Thanks for opening your first issue here! Be sure to follow the issue template!
DaanHoogland commentedon Oct 28, 2026
@rp- , can you assess this issue?
KreVet-Ka commentedon Oct 28, 2026
Additional findings from a production cluster: silent no-op detach when the live domain XML has a legacy
/dev/drbdNpathWe operate the cluster where the original report was made, and we were able to confirm the mechanism from our logs and hypervisors and assess how widespread it is.
Environment:
CloudStack 4.20.3.1 (upgraded from 4.20.3.0), KVM, libvirt 10.0.0, QEMU 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18),
Linstor primary storage (DRBD 9, hypervisors are diskless clients).
Root cause (as far as we can tell from code and logs)
Before the Linstor plugin change (linstor: Provide /dev/drbd/by-res/ resource paths to CloudStack #12300) the plugin returned
/dev/drbd{minor}as the disk path. After it, the plugin returns/dev/drbd/by-res/cs-<volume path>/0.A VM started before the upgrade keeps the old path in its live libvirt XML (
<source dev='/dev/drbd1234'/>). Live migration preserves it too (we verified this across migrations); only a stop/start regenerates the XML with the new path.On
detachVolume,KVMStorageProcessor.attachOrDetachDisklooks for the disk in the live XML by the exact path string. The strings differ even though they point to the same block device (/dev/drbd/by-res/cs-X/0is a symlink to/dev/drbdN), so the disk is not found. The agent logs the following and returns success:detach-deviceis issued, so QEMU never emitsDEVICE_DELETED. The API job succeeds, the DB setsinstance_id = NULL, but the disk stays attached in the guest and the DRBD device stays open in the QEMU process.How it manifests
The API reports success, but the disk is still attached inside the guest and in
virsh domblklist.Attaching the same volume to a VM on a different host fails (
Wrong medium type): DRBD cannot promote a second Primary whenallow-two-primariesis off for that resource.Re-attaching to the same VM fails with
target 'vdb' duplicated for disk sources '/dev/drbdN' and '/dev/drbd/by-res/…'.Attaching it to another VM on the same host succeeds, so the same DRBD device is open in two QEMU processes at once. In the reported case this window lasted roughly an hour (the stale device was later removed out-of-band from inside the guest). There is no protection against concurrent writers, so this is a potential data corruption risk. We cannot prove or rule out actual writes from the host side.
We reproduced steps 1 and 3 on a lab VM started before the upgrade (detach returned success, WARN logged, disk still present in
virsh domblklist).How widespread it is on our cluster
More than half of the running domains still carry a legacy
/dev/drbdNpath on at least one disk. All of them have a legacy root disk, i.e. they were started before the upgrade.A smaller share of those also have legacy data disks; these are the ones exposed to this bug. Root-only VMs are not exposed, since a root volume cannot be detached while the VM is running.
Kubernetes workers with CSI-managed PVCs are the worst case: the CSI driver detaches and attaches volumes automatically whenever pods move, so users cannot avoid the trigger.
Our agent logs contain several past occurrences of the WARN above (the logs we still have go back to early June), so this has been happening silently since the upgrade.
Check on a host:
virsh domblklist <domain> --details | awk '$1=="block" && $2=="disk" {print $3, $4}' | grep -E ' /dev/drbd[0-9]+$'Happy to provide any logs you might need.