apache / apache/cloudstack

Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

Open
#12,256 1 comment 0 reactions 0 assignees View on GitHub
multi-arch type:bug
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

Unable to destroy/expunge systemvm in a multihypervisor cluster/environment

### versions

ACS 4.22

### The steps to reproduce the bug

1. Create a Cloudstack environment with 2 clusters ( vmware , KVM)

2. Launch systemvm's on the vmware cluster, make sure they are in running state

3. Shutdown the vmware vcenter cluster , the system vm's will be running state , but the agent will be in a disconnected state

4. Mark the zone is disabled and try to stop or destroy the systemvm's

5. Exception observed

Image

Logs

```

2025-12-15 05:08:35,162 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-14:[ctx-63c6fb05, job-152, ctx-8ae693e9]) (logid:cb1a3aae) Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
2025-12-15 05:08:35,162 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-13:[ctx-a07f634d, job-151, ctx-1f1a2c2e]) (logid:2a920452) Unable to destroy volume [id: 9, uuid: 5113804b-2cd3-42ce-b76f-bdfa97885503] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected
at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:259)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:444)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1355)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)

```

Tried reducing the value of the expunge.delay and expunge.interval

Workaround

Manually update the database and set the removed date

mysql>update vm_instance set removed=now() where id=<>;

### What to do about it?

In order to deploy the systemvm's on the kvm cluster end user has to update the database

Provide an option to force remove the systemvm which are struck in expunging state or introduce a zone level setting which can specify the cluster for the systemvm deployment

Contributor guide

Open the contributing guide

Research direction

Start by tracing the failure through RemoteHostEndPoint.java, CloudStackPrimaryDataStoreDriverImpl.java, VolumeServiceImpl.java, and VolumeOrchestrator.java, using the disconnected-agent log as the entry point. Reproduce the two-cluster VMware/KVM scenario and inspect the destroy and expunge path. Done means system VMs can be removed or expunged without manually updating vm_instance, allowing deployment on the KVM cluster.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.