apache / apache/cloudstack

VMware VM import fails when primary storage is a datastore cluster with multiple LUNs

Ouverte
#13,885 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
component:primary-storage component:vmware type:bug
Langage dominant
Java
Étoiles
3.1k
Forks
1.4k
Merge moyen
6 j 19 h
PR mergées (30 j)
32

Description

### The required feature described as a wish

claude generated description for problem asked about in #12561

## problem

VMware "Datastore Cluster" (Storage DRS pod) primary storage pools can end up with **zero child-datastore entries** in the `storage_pool` table, even though `syncStoragePool` reports success. When this happens, importing an existing/unmanaged VM whose disk lives on one of the cluster's underlying datastores fails with:

```
Storage pool for disk Hard disk 1 (460-2000) with datastore: not found in zone ID:
```

Originally reported at: https://github.com/apache/cloudstack/discussions/12561

### Root cause (from code inspection)

Child `storage_pool` rows for a datastore-cluster pool are only created when a live `ModifyStoragePoolCommand` round-trip to a connected ESXi host returns a non-empty `datastoreClusterChildren` list:

- `plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java` (`execute(ModifyStoragePoolCommand)`, around line 5291) calls `StoragepodMO.getDatastoresInDatastoreCluster()` to enumerate child datastores.
- `vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/StoragepodMO.java` (`getDatastoresInDatastoreCluster()`, lines ~41-44) reads the `childEntity` property with **no validation or retry** — if vCenter's property collector returns an empty/partial list (e.g. permissions/RBAC restricting which datastores the CloudStack service account can see on that specific pod, or a transient inventory glitch), no exception is raised.
- `server/src/main/java/com/cloud/storage/StorageManagerImpl.java` (`syncDatastoreClusterStoragePool(...)`, lines ~2760-2815) persists whatever child list comes back. An empty list simply results in "0 added, 0 removed" — there is no check that the returned child count is non-zero or matches expectations.
- This sync path is triggered by host-connect events (`DefaultHostListener.hostConnect`, `engine/storage/volume/src/main/java/org/apache/cloudstack/storage/datastore/provider/DefaultHostListener.java`, lines ~136-186) and by the explicit `syncStoragePool` API (`StorageManagerImpl.syncStoragePool`, lines ~2643-2699), which also only samples **one arbitrary connected host** to answer.

Net effect: if the first (and every subsequent) sync attempt for this specific pod happens to get an empty answer from whichever host was asked, the pool is silently left with no child rows forever, while the API/UI report the sync as successful. Other datastore clusters in the same environment are unaffected because their sync happened to succeed.

Downstream, `getStoragePool(...)` in `server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java` (lines ~516-551) does a generic path/name substring match over all pools in the cluster/zone — it works fine for datastore-cluster children when their row exists, so this is not a separate bug in the import code; it's a downstream symptom of the missing `storage_pool` row.

## What versions of cloudstack and any infra components are you using

- CloudStack: 4.20.0.0
- Hypervisor: VMware (vCenter, ESXi)
- Primary storage: Datastore Cluster (Storage DRS pod) containing 4 LUNs
- Storage: HPE Alletra 5050 (FC/iSCSI LUNs presented as VMFS datastores)

## The steps to reproduce the bug

1. Create a VMware Datastore Cluster (Storage DRS pod) in vCenter containing multiple LUNs/datastores.
2. Add the datastore cluster as CloudStack primary storage.
3. Confirm in the `storage_pool` table that no child rows were created for the individual datastores (`parent` = the cluster pool's ID) — only the single parent/cluster row exists.
4. Run `syncStoragePool` against the pool — it completes successfully but still does not populate child rows.
5. Attempt to import an existing/unmanaged VM whose disk resides on one of the underlying datastores in that cluster.
6. Import fails: `Storage pool for disk with datastore: not found in zone ID: `.

Note: other datastore clusters in the same environment, added the same way, do have correctly populated child rows — the failure appears to depend on whether the very first sync happened to get a complete answer from vCenter.

## What to do about it?

Suggested fix directions:
- In `StoragepodMO.getDatastoresInDatastoreCluster()` / `VmwareResource.execute(ModifyStoragePoolCommand)`, treat an empty (or unexpectedly short) `childEntity`/`datastoreClusterChildren` result as suspicious rather than a normal success — e.g. log a warning, or retry against a different host before persisting.
- In `StorageManagerImpl.syncDatastoreClusterStoragePool(...)` / `syncStoragePool(...)`, surface a warning (API response and/or logs) when a datastore-cluster sync returns zero child datastores for a pool that previously had none, instead of silently reporting success.
- Consider having `syncStoragePool` try more than one connected host when the first host's answer looks incomplete, since host-level permissions/visibility to the storage pod can vary.

As a workaround, users can try re-triggering sync from a different host in the cluster, verify the vCenter service account's permissions on the affected storage pod/datastores, or remove and re-add the primary storage pool to force a fresh full discovery.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par suivre syncStoragePool et syncDatastoreClusterStoragePool dans server/src/main/java/com/cloud/storage/StorageManagerImpl.java, puis examinez le traitement de ModifyStoragePoolCommand dans VmwareResource.java et getDatastoresInDatastoreCluster() dans StoragepodMO.java. Comparez le chemin de connexion de l’hôte dans DefaultHostListener.java avec la recherche de VM non gérées dans UnmanagedVMsManagerImpl.java. Le travail est considéré comme terminé lorsqu’un résultat incomplet de cluster de datastores n’est plus accepté silencieusement et que le chemin de synchronisation/import concerné fait l’objet d’une vérification.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
cloud, infrastructure
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
48/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.