Skip to content

Powersync doesn't sync VM states after power outtage #12411

Closed
Bug
#12463
Closed
Powersync doesn't sync VM states after power outtage#12411
Bug
#12463
Assignees
DaanHoogland
Labels
component:kvmpriority: high
Milestone
4.20.3

Description

@yadvr
yadvr
opened on Jan 12, 2026

problem

After a power outtage, the host comes online and ssvm/cpvm on it start but not VRs, and the HA-enabled (via compute offering) user VMs are powered on in ACS but not running (off) on the KVM host. I'm not sure if some config is required, I expected this to work out of the box to sync VM states (i.e. set vm to stopped when it's not really running on the hypervisor host), or try to get the VM to run as it's on VM-HA enabled offering.

versions

ACS 4.22, on both x86 mini pc and arm64/raspberrypi5 KVM hosts; in adv zone using HA-enabled compute offering and uses NFS & local storage only.

The steps to reproduce the bug

...

What to do about it?

Ideally the power state of the VM should sync, and shutdown VMs on a rebooted/started KVM host after power outtage should try to bring up the user VMs (which are HA enabled).

Activity

  1. yadvr
    added
    component:kvm
    on Jan 12, 2026
  2. yadvr
    added this to the 4.22.1 milestone on Jan 12, 2026
  3. yadvr
    added the
    Bug
    issue type on Jan 12, 2026

  4. kiranchavala commented on Jan 13, 2026

    @kiranchavala
    kiranchavala
    on Jan 13, 2026
    Member

    @yadvr Not able to reproduce the issue

    1. Launched a 2 vm's one with HA enabled and another not ha enable on a KVM host 1

    2. Made sure the global setting 'commands.timeout' value to a suitable value like "CheckHealthCommand=5,CheckOnHostCommand=5"

    3. Shutdown the kvm host 1

    4. The router and HA enabled vm restarted on another kvm host

    5. The non-ha enabled vm went into stopped state

    logs of ha enabled vm

    root@acs420:~# cat  /var/log/cloudstack/management/management-server.log |grep -i "logid:48d451aa"
2026-01-13 08:38:08,653 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Processing work HAWork[577-HA-155-Running-Investigating]
2026-01-13 08:38:08,654 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) RESTART with HAWORK
2026-01-13 08:38:08,656 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) SimpleInvestigator unable to determine the state of the host.  Moving on.
2026-01-13 08:38:08,656 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) XenServerInvestigator unable to determine the state of the host.  Moving on.
2026-01-13 08:38:08,657 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,658 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,658 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,659 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5292855462067175506: Routed from 206863097656310
2026-01-13 08:38:08,659 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175506: Sending  { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,674 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175506: Received:  { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,674 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,675 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,675 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,676 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5194902170171867174: Routed from 206863097656310
2026-01-13 08:38:08,676 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867174: Sending  { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,723 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867174: Received:  { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,723 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,723 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,723 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) KVMInvestigator was able to determine host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} is in Down
2026-01-13 08:38:08,723 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA on VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,725 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds
2026-01-13 08:38:08,725 DEBUG [c.c.h.CheckOnAgentInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Unable to reach the agent for VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}: Resource [Host:6] is unreachable: Host 6: Host with specified id is not in the right state: Down
2026-01-13 08:38:08,725 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) SimpleInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,725 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) XenServerInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,727 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,728 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,728 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5292855462067175509: Routed from 206863097656310
2026-01-13 08:38:08,728 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175509: Sending  { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,771 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175509: Received:  { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,772 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5194902170171867176: Routed from 206863097656310
2026-01-13 08:38:08,772 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867176: Sending  { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,780 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867176: Received:  { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) KVMInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,780 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HypervInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,781 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) VMwareInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,781 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) testing if VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} is alive
2026-01-13 08:38:08,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command com.cloud.agent.api.PingTestCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} could not be pinged, returning that it is unknown
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Returning null since we're unable to determine state of VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,783 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) PingInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,783 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Not a System Vm, unable to determine state of VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} returning null
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Testing if VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} is alive
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} returning null
2026-01-13 08:38:08,784 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) ManagementIPSysVMInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,784 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Fencing off VM that we don't know the state of
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.FenceCommand is 1800 seconds
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5292855462067175512: Routed from 206863097656310
2026-01-13 08:38:08,786 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175512: Sending  { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-155-VM","hostGuid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","hostIp":"192.168.55.164","inSeq":"false","host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,833 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175512: Received:  { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { FenceAnswer } }
2026-01-13 08:38:08,833 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Fencer KVMFenceBuilder returned true
2026-01-13 08:38:08,833 WARN  [o.a.c.f.j.AsyncJobExecutionContext] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Job is executed without a context, setup psudo job for the executing thread
2026-01-13 08:38:08,840 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Sync job-2541 execution on object VmWorkJobQueue.155
2026-01-13 08:38:10,544 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577, ctx-99fd3c33]) (logid:48d451aa) start parameter value of enterHardwareSetup == null during processing of queued job
2026-01-13 08:38:10,548 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577, ctx-99fd3c33]) (logid:48d451aa) Sync job-2543 execution on object VmWorkJobQueue.155
2026-01-13 08:40:14,151 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA is now restarting VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} on Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:40:14,152 WARN  [c.c.a.AlertManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) alertType=[8] dataCenterId=[1] podId=[1] clusterId=[null] message=[HA starting VM: vm2-haenabled (i-2-155-VM)].
2026-01-13 08:40:14,155 WARN  [c.c.a.AlertManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [HA starting VM: vm2-haenabled (i-2-155-VM)] and content [HA starting VM: vm2-haenabled (i-2-155-VM)].
2026-01-13 08:40:14,155 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Completed work HAWork[577-HA-155-Running-Scheduled]. Took 1/5 attempts.

    Logs for Non ha enabled vm

    
root@acs420:~# cat  /var/log/cloudstack/management/management-server.log |grep -i "logid:62b9b072"
2026-01-13 08:38:08,648 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Processing work HAWork[576-HA-153-Running-Investigating]
2026-01-13 08:38:08,649 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) RESTART with HAWORK
2026-01-13 08:38:08,651 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) SimpleInvestigator unable to determine the state of the host.  Moving on.
2026-01-13 08:38:08,651 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) XenServerInvestigator unable to determine the state of the host.  Moving on.
2026-01-13 08:38:08,652 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,653 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,654 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,654 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5292855462067175505: Routed from 206863097656310
2026-01-13 08:38:08,654 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175505: Sending  { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,664 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175505: Received:  { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,664 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,664 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,665 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,665 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5194902170171867173: Routed from 206863097656310
2026-01-13 08:38:08,665 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867173: Sending  { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,676 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867173: Received:  { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,676 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,677 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,677 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) KVMInvestigator was able to determine host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} is in Down
2026-01-13 08:38:08,677 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA on VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds
2026-01-13 08:38:08,679 DEBUG [c.c.h.CheckOnAgentInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Unable to reach the agent for VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}: Resource [Host:6] is unreachable: Host 6: Host with specified id is not in the right state: Down
2026-01-13 08:38:08,679 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) SimpleInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,679 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) XenServerInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,681 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,682 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,682 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,683 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5292855462067175508: Routed from 206863097656310
2026-01-13 08:38:08,683 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175508: Sending  { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,771 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175508: Received:  { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,772 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5194902170171867177: Routed from 206863097656310
2026-01-13 08:38:08,772 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867177: Sending  { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,780 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867177: Received:  { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) KVMInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HypervInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VMwareInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) testing if VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} is alive
2026-01-13 08:38:08,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command com.cloud.agent.api.PingTestCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} could not be pinged, returning that it is unknown
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Returning null since we're unable to determine state of VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,783 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) PingInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,783 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Not a System Vm, unable to determine state of VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} returning null
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Testing if VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} is alive
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} returning null
2026-01-13 08:38:08,784 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) ManagementIPSysVMInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,784 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Fencing off VM that we don't know the state of
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.FenceCommand is 1800 seconds
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5292855462067175511: Routed from 206863097656310
2026-01-13 08:38:08,785 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175511: Sending  { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-153-VM","hostGuid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","hostIp":"192.168.55.164","inSeq":"false","host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,793 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175511: Received:  { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { FenceAnswer } }
2026-01-13 08:38:08,793 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Fencer KVMFenceBuilder returned true
2026-01-13 08:38:08,793 WARN  [o.a.c.f.j.AsyncJobExecutionContext] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Job is executed without a context, setup psudo job for the executing thread
2026-01-13 08:38:08,807 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Sync job-2539 execution on object VmWorkJobQueue.153
2026-01-13 08:38:10,517 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VM is not HA enabled so we're done.
2026-01-13 08:38:10,517 INFO  [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Completed work HAWork[576-HA-153-Running-Scheduled]. Took 1/5 attempts.

  5. yadvr commented on Jan 13, 2026

    @yadvr
    yadvr
    on Jan 13, 2026
    MemberAuthor

    @kiranchavala try to poweroff both control plane (mgmt server) and KVM host at the same time and try ?

    The commands.timeout is empty in my env, is that required to make this work?

  6. kiranchavala commented on Jan 13, 2026

    @kiranchavala
    kiranchavala
    on Jan 13, 2026
    Member

    Thanks @yadvr

    Able to reproduce the issue if both control plane (mgmt server) and KVM host are down at the same time.

    When the mgmt server and kvm host comes up later, the ha enabled vm and router says its running on kvm host but there is no vm running on the host ( virsh list --all)

    Agent log

    2026-01-13 13:04:35,239 WARN  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can not find a connection for Instance r-154-VM. Assuming the default connection.
2026-01-13 13:04:35,239 WARN  [resource.wrapper.LibvirtGetVmStatsCommandWrapper] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can't get vm stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-154-VM', continue
2026-01-13 13:04:35,240 INFO  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) No existing libvirtd connection found. Opening a new one
2026-01-13 13:04:35,240 WARN  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can not find a connection for Instance r-104-VM. Assuming the default connection.
2026-01-13 13:04:35,240 WARN  [resource.wrapper.LibvirtGetVmStatsCommandWrapper] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can't get vm stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-104-VM', continue
2026-01-13 13:04:35,241 INFO  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) No existing libvirtd connection found. Opening a new one
2026-01-13 13:04:35,241 WARN  [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can not find a connection for Instance i-2-155-VM. Assuming the default connection.
  7. yadvr
    assigned
    DaanHoogland
    on Jan 15, 2026
  8. yadvr
    added
    priority: high
    on Jan 15, 2026

  9. yadvr commented on Jan 15, 2026

    @yadvr
    yadvr
    on Jan 15, 2026
    MemberAuthor

    requesting for a higher priority issue as this is related to VM HA being broken cc @DaanHoogland

  10. sureshanaparti
    added this to Apache CloudStack 4.22.1on Feb 3, 2026
  11. sureshanaparti
    moved this to Todo in Apache CloudStack 4.22.1on Feb 3, 2026

  12. kiranchavala commented on Feb 4, 2026

    @kiranchavala
    kiranchavala
    on Feb 4, 2026
    Member

    Resolved with #12463

  13. kiranchavala
    closed this as completedon Feb 4, 2026
  14. DaanHoogland
    modified the milestones: 4.22.1, 4.20.3 on Feb 4, 2026
  15. sureshanaparti
    removed this from Apache CloudStack 4.22.1on Mar 17, 2026
  16. sureshanaparti
    linked a pull request that will close this issueVM Power state optimization #12463on Mar 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:kvmpriority: high

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions