SecondaryStorageVM don't start when using S3 as Secondary Storage
- Lenguaje dominante
- Java
- Estrellas
- 3.1k
- Forks
- 1.4k
- Merge medio
- 6 d 19 h
- PR fusionados (30 d)
- 32
Descripción
### problem
Just want to start saying that i and my company found Cloudstack to be a very interesting project and we are looking at it as a replacement for our oVirt env.
We are in the middle of a POC so this is our first cluster at the moment.
I am using S3 as secondary storage with a NFS staging connected to it.
My consoleproxy VM starts without issues but my SSVM refuse to starts and i see following error in my logs
```
2025-03-06 13:19:14,173 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-135:[ctx-9dbacb5f, job-1299/job-3426]) (logid:5e35be66) Unable to complete AsyncJobVO: {id:3426, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2
Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAES
gAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAABinQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 95037156360330, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Mar 06 13:19:09 CET 2025, re
moved: null}, job origin: 1299 java.lang.NullPointerException: Cannot invoke "String.split(String)" because "url" is null
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.addSecondaryStorageServerAddressToBuffer(SecondaryStorageManagerImpl.java:1228)
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.finalizeVirtualMachineProfile(SecondaryStorageManagerImpl.java:1216)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1272)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5467)
at jdk.internal.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
```
My guess it has to do with that the URL under Infrastructure -> Secondary storage is null? (See picture)

Even using CMK the url field is null or not even exist in the response.
```
(localcloud) 🐱 > list imagestores id=a8bbf162-cc05-4cac-92d2-f18d43627cdd filter=name,url
{
"count": 1,
"imagestore": [
{
"name": "S3"
}
]
}
```
Also if i try to enter the secondary storage and browse it i see following error log that might be connected to this.
```
2025-03-06 13:56:18,487 ERROR [c.c.a.ApiServer] (qtp652176954-348:[ctx-c959a22d, ctx-21cc0661]) (logid:6136e9bd) unhandled exception executing api command: [Ljava.lang.String;@2f1ab954 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listImageStoreObjects(StorageBrowserImpl.java:128)
at org.apache.cloudstack.api.command.admin.storage.ListImageStoreObjectsCmd.execute(ListImageStoreObjectsCmd.java:72)
```
### versions
ACS: 4.20.0
Hypervisor: KVM with Rocky 9.5
Network: OpenVswitch 3.3
Primary Storage: Ceph RBD
Secondary Storage: NetApp S3 + Ceph NFS Staging
### The steps to reproduce the bug
1. Add a S3 Secondary storage either using WebUI or CMK/API
2. Wait for templates to download and SystemVMs to start
3. ConsoleProxy starts while SSVM will not start.
### What to do about it?
Is S3 as Secondary storage stable enough to use in a prod env?
Guía de contribución
Línea de trabajo
Comienza en SecondaryStorageManagerImpl.java, alrededor de addSecondaryStorageServerAddressToBuffer (línea 1228) y finalizeVirtualMachineProfile (línea 1216), y luego inspecciona StorageBrowserImpl.java alrededor de listObjectsInStore (línea 183). Reproduce el problema con un almacén secundario S3 y la configuración indicada, comprobando la respuesta de la API image-store y el inicio de la SSVM; se considera terminado cuando la SSVM se inicia sin el error de URL null y al explorar el almacén ya no se informa de la ausencia de un endpoint remoto.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- java
- Área
- backend, cloud
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Tranquilo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 52/100