SecondaryStorageVM don't start when using S3 as Secondary Storage
- Langage dominant
- Java
- Étoiles
- 3.1k
- Forks
- 1.4k
- Merge moyen
- 6 j 19 h
- PR mergées (30 j)
- 32
Description
### problem
Just want to start saying that i and my company found Cloudstack to be a very interesting project and we are looking at it as a replacement for our oVirt env.
We are in the middle of a POC so this is our first cluster at the moment.
I am using S3 as secondary storage with a NFS staging connected to it.
My consoleproxy VM starts without issues but my SSVM refuse to starts and i see following error in my logs
```
2025-03-06 13:19:14,173 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-135:[ctx-9dbacb5f, job-1299/job-3426]) (logid:5e35be66) Unable to complete AsyncJobVO: {id:3426, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2
Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAES
gAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAABinQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 95037156360330, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Mar 06 13:19:09 CET 2025, re
moved: null}, job origin: 1299 java.lang.NullPointerException: Cannot invoke "String.split(String)" because "url" is null
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.addSecondaryStorageServerAddressToBuffer(SecondaryStorageManagerImpl.java:1228)
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.finalizeVirtualMachineProfile(SecondaryStorageManagerImpl.java:1216)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1272)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5467)
at jdk.internal.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
```
My guess it has to do with that the URL under Infrastructure -> Secondary storage is null? (See picture)

Even using CMK the url field is null or not even exist in the response.
```
(localcloud) 🐱 > list imagestores id=a8bbf162-cc05-4cac-92d2-f18d43627cdd filter=name,url
{
"count": 1,
"imagestore": [
{
"name": "S3"
}
]
}
```
Also if i try to enter the secondary storage and browse it i see following error log that might be connected to this.
```
2025-03-06 13:56:18,487 ERROR [c.c.a.ApiServer] (qtp652176954-348:[ctx-c959a22d, ctx-21cc0661]) (logid:6136e9bd) unhandled exception executing api command: [Ljava.lang.String;@2f1ab954 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listImageStoreObjects(StorageBrowserImpl.java:128)
at org.apache.cloudstack.api.command.admin.storage.ListImageStoreObjectsCmd.execute(ListImageStoreObjectsCmd.java:72)
```
### versions
ACS: 4.20.0
Hypervisor: KVM with Rocky 9.5
Network: OpenVswitch 3.3
Primary Storage: Ceph RBD
Secondary Storage: NetApp S3 + Ceph NFS Staging
### The steps to reproduce the bug
1. Add a S3 Secondary storage either using WebUI or CMK/API
2. Wait for templates to download and SystemVMs to start
3. ConsoleProxy starts while SSVM will not start.
### What to do about it?
Is S3 as Secondary storage stable enough to use in a prod env?
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez dans SecondaryStorageManagerImpl.java autour de addSecondaryStorageServerAddressToBuffer (ligne 1228) et finalizeVirtualMachineProfile (ligne 1216), puis inspectez StorageBrowserImpl.java autour de listObjectsInStore (ligne 183). Reproduisez le problème avec un stockage secondaire S3 et la configuration indiquée, en vérifiant la réponse de l’API image-store et le démarrage de la SSVM ; le travail est terminé lorsque la SSVM démarre sans l’erreur d’URL null et que la navigation dans le stockage ne signale plus l’absence d’un endpoint distant.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- backend, cloud
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 52/100