apache / apache/cloudstack

SecondaryStorageVM don't start when using S3 as Secondary Storage

Open
#10,517 5 comments 0 reactions 0 assignees View on GitHub
no-issue-activity
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

Just want to start saying that i and my company found Cloudstack to be a very interesting project and we are looking at it as a replacement for our oVirt env.
We are in the middle of a POC so this is our first cluster at the moment.

I am using S3 as secondary storage with a NFS staging connected to it.
My consoleproxy VM starts without issues but my SSVM refuse to starts and i see following error in my logs

```
2025-03-06 13:19:14,173 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-135:[ctx-9dbacb5f, job-1299/job-3426]) (logid:5e35be66) Unable to complete AsyncJobVO: {id:3426, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2
Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAES
gAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAABinQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 95037156360330, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Mar 06 13:19:09 CET 2025, re
moved: null}, job origin: 1299 java.lang.NullPointerException: Cannot invoke "String.split(String)" because "url" is null
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.addSecondaryStorageServerAddressToBuffer(SecondaryStorageManagerImpl.java:1228)
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.finalizeVirtualMachineProfile(SecondaryStorageManagerImpl.java:1216)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1272)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5467)
at jdk.internal.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

```

My guess it has to do with that the URL under Infrastructure -> Secondary storage is null? (See picture)
![Image](https://github.com/user-attachments/assets/691c9f77-7fa7-494b-8047-c846ca8ff79d)

Even using CMK the url field is null or not even exist in the response.
```
(localcloud) 🐱 > list imagestores id=a8bbf162-cc05-4cac-92d2-f18d43627cdd filter=name,url
{
"count": 1,
"imagestore": [
{
"name": "S3"
}
]
}
```

Also if i try to enter the secondary storage and browse it i see following error log that might be connected to this.
```
2025-03-06 13:56:18,487 ERROR [c.c.a.ApiServer] (qtp652176954-348:[ctx-c959a22d, ctx-21cc0661]) (logid:6136e9bd) unhandled exception executing api command: [Ljava.lang.String;@2f1ab954 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listImageStoreObjects(StorageBrowserImpl.java:128)
at org.apache.cloudstack.api.command.admin.storage.ListImageStoreObjectsCmd.execute(ListImageStoreObjectsCmd.java:72)

```

### versions

ACS: 4.20.0
Hypervisor: KVM with Rocky 9.5
Network: OpenVswitch 3.3
Primary Storage: Ceph RBD
Secondary Storage: NetApp S3 + Ceph NFS Staging

### The steps to reproduce the bug

1. Add a S3 Secondary storage either using WebUI or CMK/API
2. Wait for templates to download and SystemVMs to start
3. ConsoleProxy starts while SSVM will not start.

### What to do about it?
Is S3 as Secondary storage stable enough to use in a prod env?

Contributor guide

Open the contributing guide

Research direction

Start in SecondaryStorageManagerImpl.java around addSecondaryStorageServerAddressToBuffer (line 1228) and finalizeVirtualMachineProfile (line 1216), then inspect StorageBrowserImpl.java around listObjectsInStore (line 183). Reproduce with an S3 secondary store and the listed configuration, checking the image-store API response and SSVM startup; done means the SSVM starts without the null URL error and browsing the store no longer reports a missing remote endpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.