Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### problem

Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### versions

ACS 4.22

### The steps to reproduce the bug

1. Create a Cloudstack environment with 2 clusters ( vmware , KVM)

2. Launch systemvm's on the vmware cluster, make sure they are in running state

3. Shutdown the vmware vcenter cluster , the system vm's will be running state , but the agent will be in a disconnected state

4. Mark the zone is disabled and try to stop or destroy the systemvm's

5. Exception observed

Logs

```

2025-12-15 05:08:35,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-14:[ctx-63c6fb05, job-152, ctx-8ae693e9]) (logid:cb1a3aae) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Unable to destroy volume [id: 9, uuid: 5113804b-2cd3-42ce-b76f-bdfa97885503] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:259)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:444)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1355)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

```

Tried reducing the value of the expunge.delay and expunge.interval

Workaround

Manually update the database and set the removed date

mysql>update vm_instance set removed=now() where id=<>;

### What to do about it?

In order to deploy the systemvm's on the kvm cluster end user has to update the database

Provide an option to force remove the systemvm which are struck in expunging state or introduce a zone level setting which can specify the cluster for the systemvm deployment

Bắt đầu bằng cách lần theo lỗi qua RemoteHostEndPoint.java, CloudStackPrimaryDataStoreDriverImpl.java, VolumeServiceImpl.java và VolumeOrchestrator.java, sử dụng log của agent bị ngắt kết nối làm điểm bắt đầu. Tái hiện kịch bản VMware/KVM với hai cluster và kiểm tra đường dẫn destroy và expunge. Công việc được hoàn tất khi có thể xóa hoặc expunge system VMs mà không cần cập nhật thủ công vm_instance, cho phép triển khai trên cluster KVM.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.