Unable to remove vcenter/esxi host from cloudstack with force/clean if the vcenter is deleted externally

### problem

Unable to remove vcenter/esxi host from cloudstack with force/cleanly if the vcenter is deleted externally

### versions

ACS 4.22

### The steps to reproduce the bug

1. Set up Cloudstack env with a vcenter

2. Vcenter gets deleted externally

3. Admin user goes to the Ui > Zones > Remove vmware details from zone

Exception :

Logs

```
2026-05-18 06:08:03,827 DEBUG [c.c.a.ApiServlet] (qtp2038105753-25:[ctx-04dda2ab]) (logid:f6e52355) ===START=== 10.0.3.251 -- POST
command=removeVmwareDc
response=json
zoneid=7696337c-f532-41a9-a37a-a974b6fff240
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:08:03,827 DEBUG [c.c.a.ApiServlet] (qtp2038105753-25:[ctx-04dda2ab]) (logid:f6e52355) Two factor authentication is already verified for the user 2, so skipping
2026-05-18 06:08:03,832 DEBUG [c.c.a.ApiServer] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-05-18 06:08:03,834 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed.
2026-05-18 06:08:03,834 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-05-18 06:08:03,834 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-05-18 06:08:03,840 WARN [o.a.c.a.c.a.z.RemoveVmwareDcCmd] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) The zone has one or more resources (like cluster), hence not able to remove VMware datacenter from zone. Please remove all resource from zone, and retry. Exception: com.cloud.exception.ResourceInUseException: Zone has one or more clusters. Can't remove VMware datacenter to zone which already has clusters.
at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.validateZoneWithResources(VmwareManagerImpl.java:1433)
at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.removeVmwareDatacenter(VmwareManagerImpl.java:1350)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

```
4. Next admin tries to delete the host in the vmware cluster from cloudstack by keeping the host in maintenance mode. There is no option to force delete the host

logs

```
2026-05-18 06:14:03,175 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b]) (logid:3f7a543d) ===START=== 10.0.3.251 -- POST
command=prepareHostForMaintenance
response=json
id=4a0c6597-86e8-4409-adfe-63970f27ca09
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:14:03,175 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b]) (logid:3f7a543d) Two factor authentication is already verified for the user 2, so skipping
2026-05-18 06:14:03,179 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-05-18 06:14:03,180 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed.
2026-05-18 06:14:03,180 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-05-18 06:14:03,180 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-05-18 06:14:03,193 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) Retrieved cmdEventType from job info: MAINT.PREPARE
2026-05-18 06:14:03,195 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-33:[ctx-1f2955e7, job-41]) (logid:1e20138f) Add job-41 into job monitoring
2026-05-18 06:14:03,197 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) submit async job-41, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"QAZorHbuhisqM_pl6NxnR1wC3b8\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"154\",\"id\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"4a0c6597-86e8-4409-adfe-63970f27ca09\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"cmdEventType\":\"MAINT.PREPARE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":41,"initMsid":32989190816243,"instanceId":1,"instanceType":"Host","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8aa5c307-73fc-4cad-bd3b-5caaeafb483a"}
2026-05-18 06:14:03,198 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) ===END=== 10.0.3.251 -- POST
command=prepareHostForMaintenance
response=json
id=4a0c6597-86e8-4409-adfe-63970f27ca09
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:14:03,198 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-33:[ctx-1f2955e7, job-41]) (logid:8aa5c307) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"QAZorHbuhisqM_pl6NxnR1wC3b8\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"154\",\"id\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"4a0c6597-86e8-4409-adfe-63970f27ca09\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"cmdEventType\":\"MAINT.PREPARE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":41,"initMsid":32989190816243,"instanceId":1,"instanceType":"Host","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8aa5c307-73fc-4cad-bd3b-5caaeafb483a"}
2026-05-18 06:14:03,213 INFO [c.c.r.ResourceManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Maintenance: attempting maintenance of host Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"}
2026-05-18 06:14:03,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Can not send command com.cloud.agent.api.MaintainCommand due to Host Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"} not being up
2026-05-18 06:14:03,216 WARN [c.c.r.ResourceManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Unable to send MaintainCommand to host: Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"}

```

5. Next admin tries to delete the primary storage associated with the VMware cluster

logs

```
2026-05-18 06:18:34,608 DEBUG [c.c.s.StorageManagerImpl] (qtp2038105753-23:[ctx-eef67450, ctx-7f6db4e9]) (logid:2da339a0) Cannot delete storage pool StoragePool {"id":1,"name":"ref-trl-11757-v-Mol9-harikrishna-patnala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"4ac4c575-ace8-391c-892a-b207bc1248c6"} as the following non-destroyed volumes are on it: [Volume [36d6c20e-33f6-4403-a4fb-64efa7e2cdde] (attached to VM [4195861e-b559-40fc-8829-61217d75d9da])].
2026-05-18 06:18:34,608 ERROR [c.c.a.ApiServer] (qtp2038105753-23:[ctx-eef67450, ctx-7f6db4e9]) (logid:2da339a0) unhandled exception executing api command: [Ljava.lang.String;@7c5be73a com.cloud.utils.exception.CloudRuntimeException: Cannot delete pool StoragePool {"id":1,"name":"ref-trl-11757-v-Mol9-harikrishna-patnala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"4ac4c575-ace8-391c-892a-b207bc1248c6"} as there are non-destroyed volumes associated to this pool.
at com.cloud.storage.StorageManagerImpl.deleteDataStoreInternal(StorageManagerImpl.java:1765)

```

6. Admin user tries to expunge the vm which is having volumes in the primary storage

logs

```
2026-05-18 06:20:16,910 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Cleaning storage for VM [VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Expunging","type":"User","uuid":"4195861e-b559-40fc-8829-61217d75d9da"}].
2026-05-18 06:20:16,911 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Skipping destroy for the volume [{"name":"ROOT-3","uuid":"36d6c20e-33f6-4403-a4fb-64efa7e2cdde"}] as it is in [Destroy] state.
2026-05-18 06:20:16,931 DEBUG [c.c.h.XenServerGuru] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) We are returning the default host to execute commands because the command is not of Copy type.
2026-05-18 06:20:16,931 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2026-05-18 06:20:16,932 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Failed to send command, due to Agent:2, com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Host with specified id is not in the right state: Alert
2026-05-18 06:20:16,932 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Unable to destroy volume [id: 3, uuid: 36d6c20e-33f6-4403-a4fb-64efa7e2cdde] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:2, com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Host with specified id is not in the right state: Alert
at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:260)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:459)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1357)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

```

7. User Tries to destroy the volume

Logs

```
2026-05-18 06:24:41,199 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028]) (logid:ffa4161f) ===START=== 10.0.3.251 -- POST
command=deleteVolume
response=json
id=36d6c20e-33f6-4403-a4fb-64efa7e2cdde
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:24:41,199 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028]) (logid:ffa4161f) Two factor authentication is already verified for the user 2, so skipping
2026-05-18 06:24:41,204 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-05-18 06:24:41,206 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed.
2026-05-18 06:24:41,206 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-05-18 06:24:41,206 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-05-18 06:24:41,213 DEBUG [c.c.u.AccountManagerImpl] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Account [Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]] has access to resource.
2026-05-18 06:24:41,229 INFO [c.c.a.ApiServer] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Please specify a volume that is not attached to any VM.
2026-05-18 06:24:41,230 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) ===END=== 10.0.3.251 -- POST
command=deleteVolume
response=json
id=36d6c20e-33f6-4403-a4fb-64efa7e2cdde
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8
```

Admin user has to manually execute the DB queries to remove the vcenter details from the cloudstack zone

### What to do about it?

Expected behaviour

Provide an option of force removing the vcenter details from cloudstack if the vcenter is destroyed externally and is unrecoverable

Inizia con VmwareManagerImpl.validateZoneWithResources e removeVmwareDatacenter, quindi traccia i percorsi di pulizia di host, storage e volumi mostrati in ResourceManagerImpl, StorageManagerImpl e VolumeOrchestrator. Riproduci lo scenario di un vCenter eliminato esternamente e verifica che le risorse VMware possano essere rimosse o ripulite senza richiedere host non disponibili o volumi non distrutti.

Scritto dal modello di indicizzazione a partire dal testo della issue.