Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### problem

Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### versions

ACS 4.22

### The steps to reproduce the bug

1. Create a Cloudstack environment with 2 clusters ( vmware , KVM)

2. Launch systemvm's on the vmware cluster, make sure they are in running state

3. Shutdown the vmware vcenter cluster , the system vm's will be running state , but the agent will be in a disconnected state

4. Mark the zone is disabled and try to stop or destroy the systemvm's

5. Exception observed

Logs

```

2025-12-15 05:08:35,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-14:[ctx-63c6fb05, job-152, ctx-8ae693e9]) (logid:cb1a3aae) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Unable to destroy volume [id: 9, uuid: 5113804b-2cd3-42ce-b76f-bdfa97885503] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:259)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:444)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1355)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

```

Tried reducing the value of the expunge.delay and expunge.interval

Workaround

Manually update the database and set the removed date

mysql>update vm_instance set removed=now() where id=<>;

### What to do about it?

In order to deploy the systemvm's on the kvm cluster end user has to update the database

Provide an option to force remove the systemvm which are struck in expunging state or introduce a zone level setting which can specify the cluster for the systemvm deployment

Beginnen Sie damit, den Fehler durch RemoteHostEndPoint.java, CloudStackPrimaryDataStoreDriverImpl.java, VolumeServiceImpl.java und VolumeOrchestrator.java zurückzuverfolgen, wobei das Protokoll des getrennten Agenten als Einstiegspunkt dient. Reproduzieren Sie das VMware/KVM-Szenario mit zwei Clustern und untersuchen Sie den Destroy- und Expunge-Pfad. Erledigt ist die Aufgabe, wenn System-VMs entfernt oder expungiert werden können, ohne vm_instance manuell zu aktualisieren, sodass eine Bereitstellung auf dem KVM-Cluster möglich ist.

Vom Indexierungsmodell aus dem Issue-Text verfasst.