CloudStack 4.22: `prepareHostForMaintenance` throws NPE when stale destroyed volume references removed storage pool
- Vorherrschende Sprache
- Java
- Sterne
- 3.1k
- Forks
- 1.4k
- Ø Merge
- 6 T. 19 Std.
- Gemergte PRs (30 T.)
- 32
Beschreibung
### problem
## Description
While attempting to place a KVM host into Maintenance mode in CloudStack 4.22, the maintenance operation failed with a `NullPointerException`.
The issue appears to occur when a VM has a stale/destroyed volume entry in the `volumes` table that still references a removed storage pool.
Instead of gracefully ignoring the stale volume metadata or returning a user-facing validation error, CloudStack crashes during maintenance preparation.
## Environment
* CloudStack version: 4.22
* Hypervisor: KVM
* Primary storage: NFS
* Database: MySQL/MariaDB
## Steps to reproduce
1. Have a VM with:
* one valid active ROOT volume
* one stale/destroyed ROOT volume entry still present in the `volumes` table
2. The stale volume references a storage pool which has already been removed.
3. Attempt to place the host running the VM into Maintenance mode.
Example problematic DB state:
### VM
```sql
SELECT id, uuid, name, instance_id, pool_id, state, removed
FROM volumes
WHERE instance_id = 446
ORDER BY id;
```
Result:
```text
+------+--------------------------------------+----------+-------------+---------+---------+---------+
| id | uuid | name | instance_id | pool_id | state | removed |
+------+--------------------------------------+----------+-------------+---------+---------+---------+
| 928 | 6dea3d6f-bd6d-4e8b-9524-6e99c029694c | ROOT-446 | 446 | 4 | Destroy | NULL |
| 1554 | 80acf9ae-b047-41b8-bded-cdceb6de7051 | ROOT-446 | 446 | 2 | Ready | NULL |
+------+--------------------------------------+----------+-------------+---------+---------+---------+
```
### Storage pool state
The stale volume references storage pool ID 4, which is already removed:
```text
storage_pool_name: Export-Domain
storage_pool_removed: 2026-04-23 13:44:08
```
## Actual result
Host maintenance fails with:
```text
java.lang.NullPointerException: Cannot invoke
"org.apache.cloudstack.storage.datastore.db.StoragePoolVO.isLocal()"
because "storagePool" is null
```
Relevant stack trace:
```text
at com.cloud.vm.UserVmManagerImpl.isAnyVmVolumeUsingLocalStorage(UserVmManagerImpl.java:7558)
at com.cloud.vm.UserVmManagerImpl.isVMUsingLocalStorage(UserVmManagerImpl.java:7121)
at com.cloud.resource.ResourceManagerImpl.doMaintain(ResourceManagerImpl.java:1553)
at com.cloud.resource.ResourceManagerImpl.maintain(ResourceManagerImpl.java:1653)
at org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd.execute(PrepareForHostMaintenanceCmd.java:99)
```
## Expected result
CloudStack should not throw an unhandled `NullPointerException`.
Possible expected behavior:
* ignore destroyed/removed stale volumes during maintenance evaluation
* skip volumes attached to removed pools
* or return a proper validation error identifying the problematic VM/volume
## Workaround
Marking the stale destroyed volume row as removed allowed maintenance to proceed:
```sql
UPDATE volumes
SET removed = NOW()
WHERE id = 928
AND instance_id = 446
AND state = 'Destroy'
AND removed IS NULL
AND pool_id = 4;
```
## Additional notes
The issue appears to be triggered specifically by:
* stale destroyed volume rows
* still linked to an active/running VM
* referencing removed storage pools
* while evaluating VM local-storage usage during host maintenance
### versions
The versions ACS 4.22, KVM (should not be relevant)
### The steps to reproduce the bug
1.
2.
3.
...
### What to do about it?
_No response_
Beitragsleitfaden
Rechercherichtung
Beginne bei UserVmManagerImpl.isAnyVmVolumeUsingLocalStorage in Zeile 7558 und verfolge seine Aufrufer in isVMUsingLocalStorage und ResourceManagerImpl.doMaintain. Stelle den Zustand eines veralteten, zerstörten Volumes und eines entfernten Storage-Pools nach und überprüfe anschließend, dass die Wartung keinen unbehandelten NullPointerException mehr auslöst und dass das gewählte Verhalten für das veraltete Volume durch einen passenden Test abgedeckt ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java, mariadb, mysql
- Bereich
- backend, cloud, infrastructure
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 55/100