VM migration fails with "Domain not found" and ambiguous libvirt connection handling

### problem

We are experiencing a VM migration failure in CloudStack where the migration process fails with a "Domain not found" error on the source host.

During migration, CloudStack reports "Migration was unsuccessful" without detailed error information on the management server. However, on the source host (KVM/libvirt), the agent fails to locate the VM domain during execution of the MigrateCommand.

**log management-server
`2026-04-21 15:21:29,314 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp652176954-26811:[ctx-82bf6948, ctx-c49af74b]) (logid:8f7aa9da) submit async job-303869, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd","cmdInfo":"{\"virtualmachineid\":\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"xxx\",\"hostid\":\"2995de01-4d98-49c9-b379-ff8adf0f0722\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"6185591\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"2995de01-4d98-49c9-b379-ff8adf0f0722\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\\\"}\",\"ctxAccountId\":\"2\",\"cmdEventType\":\"VM.MIGRATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":303869,"initMsid":345050754440,"instanceId":4127,"instanceType":"VirtualMachine","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"569f38b1-bc6c-422c-a4ee-5fc8e3ebb7c3"}
2026-04-21 15:21:29,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-57:[ctx-964ae7d1, job-303869]) (logid:569f38b1) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd","cmdInfo":"{\"virtualmachineid\":\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"xxx\",\"hostid\":\"2995de01-4d98-49c9-b379-ff8adf0f0722\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"6185591\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"2995de01-4d98-49c9-b379-ff8adf0f0722\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\\\"}\",\"ctxAccountId\":\"2\",\"cmdEventType\":\"VM.MIGRATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":303869,"initMsid":345050754440,"instanceId":4127,"instanceType":"VirtualMachine","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"569f38b1-bc6c-422c-a4ee-5fc8e3ebb7c3"}
2026-04-21 15:21:29,353 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) Checking if there are any ongoing snapshots volumes associated with VM VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:29,354 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) Found 1 no. of volumes of type null for vm with VM ID VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:29,355 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) The count of ongoing Snapshots for VM VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} and disk type null is 0
2026-04-21 15:21:29,355 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) Found no ongoing snapshots on volumes associated with the vm VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:30,999 INFO [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Migrating VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} to Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(2)-Pod(2)-Cluster(6)-Host(324)-Storage()]
2026-04-21 15:21:31,022 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Preparing to migrate [1] volumes for VM [VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}].
2026-04-21 15:21:31,106 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Seq 446-3275242829005214246: Forwarding Seq 324-3275242829005214246: { Cmd , MgmtId: 345050754440, via: 324(com11), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00 GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other Ubuntu (64-bit)","platformEmulator":"Other Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00 GB) 21474836480","path":"/dev/storpool-byid/","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"rollback":"false","wait":"0","bypassHostMaintenance":"false"}}] } to 345050719314
2026-04-21 15:21:31,368 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} state transited from [Running] to [Migrating] with event [MigrationRequested]. VM's original host: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}, new host: Host {"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}, host before state transition: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}
2026-04-21 15:21:31,396 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Setting VLAN persistence to [{"17742":true}] as part of migrate command for VM [VM {id: "4127", name: "i-2-4127-VM", uuid: "bf5f467f-c17a-4e6e-b3e2-1beb97941324", type: "User"}].
2026-04-21 15:21:31,396 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Setting CPU shares to [800] as part of migrate command for VM [VM {id: "4127", name: "i-2-4127-VM", uuid: "bf5f467f-c17a-4e6e-b3e2-1beb97941324", type: "User"}].
2026-04-21 15:21:31,399 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Seq 446-8848728842852947950: Forwarding Seq 318-8848728842852947950: { Cmd , MgmtId: 345050754440, via: 318(com10), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-4127-VM","destinationIp":"10.xx.xx.61","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"dca8e648-8706-3504-adb2-7f20ea59398c-LibvirtComputingResource","windows":"false","virtualMachine":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00 GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other Ubuntu (64-bit)","platformEmulator":"Other Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00 GB) 21474836480","path":"/dev/storpool-byid/","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"xxx","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"800","vlanToPersistenceMap":{"17742":"true"},"wait":"0","bypassHostMaintenance":"false"}}] } to 345050719314
2026-04-21 15:21:31,447 INFO [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Migration was unsuccessful. Cleaning up: VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}
2026-04-21 15:21:31,454 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Releasing [1] volumes for VM [VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}] from host [Host {"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}].
2026-04-21 15:21:31,463 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} is a [User], returning null for control Nic IP.
2026-04-21 15:21:31,482 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} state transited from [Migrating] to [Running] with event [OperationFailed]. VM's original host: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}, new host: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}, host before state transition: Host {"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}`

### Additional Context

From the libvirt/qemu logs on the source host, it is observed that the VM domain **did exist previously** as part of an earlier migration.

At around 15:20, the VM was successfully started on the source host and participated in a migration process:

**log cloudstack-agent (source host)
`2026-04-21 15:21:29,362 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:569f38b1) Processing command: com.cloud.agent.api.GetVmDiskStatsCommand
2026-04-21 15:21:29,362 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-3:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system
2026-04-21 15:21:29,374 WARN [resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper] (AgentRequest-Handler-3:[]) (logid:569f38b1) Can't get vm disk stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM', continue
2026-04-21 15:21:29,375 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:569f38b1) Seq 318-8848728842852947947: { Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetVmDiskStatsAnswer":{"hostName":"com10","vmDiskStatsMap":{},"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] }
2026-04-21 15:21:29,422 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:569f38b1) Processing command: com.cloud.agent.api.GetVmNetworkStatsCommand
2026-04-21 15:21:29,422 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system
2026-04-21 15:21:29,435 WARN [resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper] (AgentRequest-Handler-2:[]) (logid:569f38b1) Can't get vm network stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM', continue
2026-04-21 15:21:29,436 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:569f38b1) Seq 318-8848728842852947948: { Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetVmNetworkStatsAnswer":{"hostName":"com10","vmNetworkStatsMap":{},"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] }
2026-04-21 15:21:31,403 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:569f38b1) Request:Seq 318-8848728842852947950: { Cmd , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-4127-VM","destinationIp":"10.120.25.61","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"dca8e648-8706-3504-adb2-7f20ea59398c-LibvirtComputingResource","windows":"false","virtualMachine":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00 GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other Ubuntu (64-bit)","platformEmulator":"Other Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"xxxx-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00 GB) 21474836480","path":"/dev/storpool-byid/","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"800","vlanToPersistenceMap":{"17742":"true"},"wait":"0","bypassHostMaintenance":"false"}}] }
2026-04-21 15:21:31,403 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:569f38b1) Processing command: com.cloud.agent.api.MigrateCommand
2026-04-21 15:21:31,419 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:569f38b1) Trying to migrate VM [i-2-4127-VM] to destination host: [qemu+tls://10.120.25.61/system].
2026-04-21 15:21:31,419 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:569f38b1) Property [hypervisor.uri] has empty or null value. Using default value [null].
2026-04-21 15:21:31,419 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system
2026-04-21 15:21:31,431 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find KVM connection for Instance: i-2-4127-VM, continuing.
2026-04-21 15:21:31,432 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:569f38b1) Property [hypervisor.uri] has empty or null value. Using default value [null].
2026-04-21 15:21:31,432 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection at: lxc:///
2026-04-21 15:21:31,432 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) No existing libvirtd connection found. Opening a new one
2026-04-21 15:21:31,432 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:569f38b1) Property [libvirt.events.enabled] has empty or null value. Using default value [false].
2026-04-21 15:21:31,432 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Libvirt event listening is disabled, not setting up event loop
2026-04-21 15:21:31,433 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find LXC connection for Instance: i-2-4127-VM, continuing.
2026-04-21 15:21:31,433 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find a connection for Instance i-2-4127-VM. Assuming the default connection.
2026-04-21 15:21:31,433 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system
2026-04-21 15:21:31,445 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:569f38b1) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM'
2026-04-21 15:21:31,446 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:569f38b1) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM'
2026-04-21 15:21:31,446 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:569f38b1) Found domain with name [i-2-4127-VM]. Starting VM migration to host [qemu+tls://10.120.25.61/system].
2026-04-21 15:21:31,446 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can't migrate domain [i-2-4127-VM] due to: [Domain not found: no domain with matching name 'i-2-4127-VM']. org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM'
2026-04-21 15:21:31,447 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:569f38b1) Seq 318-8848728842852947950: { Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10, [{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Domain not found: no domain with matching name 'i-2-4127-VM'","wait":"0","bypassHostMaintenance":"false"}}] }`

**log libvirt (source host)
`2026-04-21 08:20:25.095+0000: starting up libvirt version: 8.0.0, package: 23.2.module_el8.10.0+3869+b8959270 (AlmaLinux Packaging Team , 2024-07-10-04:57:33, ), qemu version: 6.2.0qemu-kvm-6.2.0-53.module_el8.10.0+3897+eb84924d, kernel: 4.18.0-553.22.1.el8_10.x86_64, hostname: com10
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.config \
/usr/libexec/qemu-kvm \
-name guest=i-2-4127-VM,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/master-key.aes"}' \
-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on \
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
-uuid bf5f467f-c17a-4e6e-b3e2-1beb97941324 \
-smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=bf5f467f-c17a-4e6e-b3e2-1beb97941324,uuid=bf5f467f-c17a-4e6e-b3e2-1beb97941324' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=70,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev '{"driver":"host_device","filename":"/dev/storpool-byid/","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c9e66d60139c4e90ab9a \
-device ide-cd,bus=ide.1,unit=0,id=ide0-1-0,bootindex=1 \
-netdev tap,fd=76,id=hostnet0,vhost=on,vhostfd=86 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:02:03:0f:00:3c,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=67,server=on,wait=off \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"none"}' \
-object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \
-vnc 10.xx.xx.60:0,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device i6300esb,id=watchdog0,bus=pci.0,addr=0x7 \
-watchdog-action none \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2026-04-21 08:20:25.095+0000: Domain id=21617 is tainted: high-privileges
2026-04-21 08:20:25.096+0000: Domain id=21617 is tainted: hook-script
char device redirected to /dev/pts/25 (label charserial0)
2026-04-21T08:20:25.719385Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2026-04-21 08:20:56.634+0000: initiating migration
2026-04-21 08:20:57.921+0000: shutting down, reason=migrated
2026-04-21T08:20:57.922592Z qemu-kvm: terminating on signal 15 from pid 3930964 (/usr/sbin/libvirtd)
`

**log libvirt (dest host)
`2026-04-21 08:20:54.470+0000: starting up libvirt version: 8.0.0, package: 23.2.module_el8.10.0+3869+b8959270 (AlmaLinux Packaging Team , 2024-07-10-04:57:33, ), qemu version: 6.2.0qemu-kvm-6.2.0-53.module_el8.10.0+3897+eb84924d, kernel: 4.18.0-553.22.1.el8_10.x86_64, hostname: com11
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.config \
/usr/libexec/qemu-kvm \
-name guest=i-2-4127-VM,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/master-key.aes"}' \
-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on \
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
-uuid bf5f467f-c17a-4e6e-b3e2-1beb97941324 \
-smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=bf5f467f-c17a-4e6e-b3e2-1beb97941324,uuid=bf5f467f-c17a-4e6e-b3e2-1beb97941324' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=42,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev '{"driver":"host_device","filename":"/dev/storpool-byid/","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c9e66d60139c4e90ab9a \
-device ide-cd,bus=ide.1,unit=0,id=ide0-1-0,bootindex=1 \
-netdev tap,fd=46,id=hostnet0,vhost=on,vhostfd=55 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:02:03:0f:00:3c,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=41,server=on,wait=off \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-audiodev '{"id":"audio1","driver":"none"}' \
-object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \
-vnc 10.xx.xx.61:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device i6300esb,id=watchdog0,bus=pci.0,addr=0x7 \
-watchdog-action none \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2026-04-21 08:20:54.470+0000: Domain id=321 is tainted: high-privileges
2026-04-21 08:20:54.470+0000: Domain id=321 is tainted: hook-script
char device redirected to /dev/pts/3 (label charserial0)
2026-04-21T08:20:55.588925Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2026-04-21T08:21:32.547875Z qemu-kvm: terminating on signal 15 from pid 1223112 (/usr/sbin/libvirtd)
2026-04-21 08:21:32.806+0000: shutting down, reason=shutdown`

- After the failed migration:
- The VM is **not running on the source host**
- The VM is also **not running on the destination host**

### Additional Observation

In some cases, the same error:
"Domain not found: no domain with matching name"
does not always result in a failed migration.

There are scenarios where:
- The VM is successfully migrated at the KVM/libvirt level
- But the CloudStack database is not updated to the new host

As a result:
- CloudStack still considers the VM on the source host
- When starting the VM, it may run again on the source host while already running on the destination host

This leads to:
- **Duplicate VM running on two hosts**
- Potential **split-brain condition**

### versions

CloudStack version: 4.22.0.0
Hypervisor: KVM (libvirt 8.0.0, qemu-kvm 6.2.0)
OS: AlmaLinux / RHEL-based

### The steps to reproduce the bug

1. Perform VM migration sequentially across multiple hosts (e.g., com01 → com02 → com03 → … → com24)
2. Repeat the migration process multiple times across the cluster until the issue occurs
3. this problem occurs when migrating from com10 to com11

### What to do about it?
1. Ensure VM domain existence is properly validated on the source host before executing migration
2. Improve migration handling when the domain is missing:
- Fail early with a clear and explicit error
- Avoid proceeding with migration when the domain cannot be found
3. Improve logging and error reporting:
- Provide detailed failure reason from agent/libvirt to management server
- Ensure logs are consistent and not misleading

Comienza con la ruta de MigrateCommand en el agente KVM/libvirt y compárala con los registros de migración del management-server proporcionados en el issue. Traza cómo el host de origen resuelve el dominio de la VM y cómo se notifican los fallos; el trabajo estará terminado cuando la migración ya no falle con "Domain not found" y el management-server exponga el error relevante.

Escrito por el modelo de indexación a partir del texto del issue.