apache / apache/cloudstack

CloudStack 4.22: `prepareHostForMaintenance` throws NPE when stale destroyed volume references removed storage pool

Offen
#13,124 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.4k
Ø Merge
6 T. 19 Std.
Gemergte PRs (30 T.)
32

Beschreibung

### problem

## Description

While attempting to place a KVM host into Maintenance mode in CloudStack 4.22, the maintenance operation failed with a `NullPointerException`.

The issue appears to occur when a VM has a stale/destroyed volume entry in the `volumes` table that still references a removed storage pool.

Instead of gracefully ignoring the stale volume metadata or returning a user-facing validation error, CloudStack crashes during maintenance preparation.

## Environment

* CloudStack version: 4.22
* Hypervisor: KVM
* Primary storage: NFS
* Database: MySQL/MariaDB

## Steps to reproduce

1. Have a VM with:

* one valid active ROOT volume
* one stale/destroyed ROOT volume entry still present in the `volumes` table
2. The stale volume references a storage pool which has already been removed.
3. Attempt to place the host running the VM into Maintenance mode.

Example problematic DB state:

### VM

```sql
SELECT id, uuid, name, instance_id, pool_id, state, removed
FROM volumes
WHERE instance_id = 446
ORDER BY id;
```

Result:

```text
+------+--------------------------------------+----------+-------------+---------+---------+---------+
| id | uuid | name | instance_id | pool_id | state | removed |
+------+--------------------------------------+----------+-------------+---------+---------+---------+
| 928 | 6dea3d6f-bd6d-4e8b-9524-6e99c029694c | ROOT-446 | 446 | 4 | Destroy | NULL |
| 1554 | 80acf9ae-b047-41b8-bded-cdceb6de7051 | ROOT-446 | 446 | 2 | Ready | NULL |
+------+--------------------------------------+----------+-------------+---------+---------+---------+
```

### Storage pool state

The stale volume references storage pool ID 4, which is already removed:

```text
storage_pool_name: Export-Domain
storage_pool_removed: 2026-04-23 13:44:08
```

## Actual result

Host maintenance fails with:

```text
java.lang.NullPointerException: Cannot invoke
"org.apache.cloudstack.storage.datastore.db.StoragePoolVO.isLocal()"
because "storagePool" is null
```

Relevant stack trace:

```text
at com.cloud.vm.UserVmManagerImpl.isAnyVmVolumeUsingLocalStorage(UserVmManagerImpl.java:7558)
at com.cloud.vm.UserVmManagerImpl.isVMUsingLocalStorage(UserVmManagerImpl.java:7121)
at com.cloud.resource.ResourceManagerImpl.doMaintain(ResourceManagerImpl.java:1553)
at com.cloud.resource.ResourceManagerImpl.maintain(ResourceManagerImpl.java:1653)
at org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd.execute(PrepareForHostMaintenanceCmd.java:99)
```

## Expected result

CloudStack should not throw an unhandled `NullPointerException`.

Possible expected behavior:

* ignore destroyed/removed stale volumes during maintenance evaluation
* skip volumes attached to removed pools
* or return a proper validation error identifying the problematic VM/volume

## Workaround

Marking the stale destroyed volume row as removed allowed maintenance to proceed:

```sql
UPDATE volumes
SET removed = NOW()
WHERE id = 928
AND instance_id = 446
AND state = 'Destroy'
AND removed IS NULL
AND pool_id = 4;
```

## Additional notes

The issue appears to be triggered specifically by:

* stale destroyed volume rows
* still linked to an active/running VM
* referencing removed storage pools
* while evaluating VM local-storage usage during host maintenance

### versions

The versions ACS 4.22, KVM (should not be relevant)

### The steps to reproduce the bug

1.
2.
3.
...

### What to do about it?

_No response_

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne bei UserVmManagerImpl.isAnyVmVolumeUsingLocalStorage in Zeile 7558 und verfolge seine Aufrufer in isVMUsingLocalStorage und ResourceManagerImpl.doMaintain. Stelle den Zustand eines veralteten, zerstörten Volumes und eines entfernten Storage-Pools nach und überprüfe anschließend, dass die Wartung keinen unbehandelten NullPointerException mehr auslöst und dass das gewählte Verhalten für das veraltete Volume durch einen passenden Test abgedeckt ist.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java, mariadb, mysql
Bereich
backend, cloud, infrastructure
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
55/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.