Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### problem

Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### versions

ACS 4.22

### The steps to reproduce the bug

1. Create a Cloudstack environment with 2 clusters ( vmware , KVM)

2. Launch systemvm's on the vmware cluster, make sure they are in running state

3. Shutdown the vmware vcenter cluster , the system vm's will be running state , but the agent will be in a disconnected state

4. Mark the zone is disabled and try to stop or destroy the systemvm's

5. Exception observed

Logs

```

2025-12-15 05:08:35,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-14:[ctx-63c6fb05, job-152, ctx-8ae693e9]) (logid:cb1a3aae) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Unable to destroy volume [id: 9, uuid: 5113804b-2cd3-42ce-b76f-bdfa97885503] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:259)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:444)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1355)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

```

Tried reducing the value of the expunge.delay and expunge.interval

Workaround

Manually update the database and set the removed date

mysql>update vm_instance set removed=now() where id=<>;

### What to do about it?

In order to deploy the systemvm's on the kvm cluster end user has to update the database

Provide an option to force remove the systemvm which are struck in expunging state or introduce a zone level setting which can specify the cluster for the systemvm deployment

Inizia tracciando il problema attraverso RemoteHostEndPoint.java, CloudStackPrimaryDataStoreDriverImpl.java, VolumeServiceImpl.java e VolumeOrchestrator.java, usando il log dell’agent disconnesso come punto di ingresso. Riproduci lo scenario VMware/KVM con due cluster e ispeziona il percorso di destroy ed expunge. Il lavoro è completato quando le system VMs possono essere rimosse o sottoposte a expunge senza aggiornare manualmente vm_instance, consentendo il deployment sul cluster KVM.

Scritto dal modello di indicizzazione a partire dal testo della issue.