apache / apache/cloudstack

SecondaryStorageVM don't start when using S3 as Secondary Storage

Offen
#10,517 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
no-issue-activity
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.4k
Ø Merge
6 T. 19 Std.
Gemergte PRs (30 T.)
32

Beschreibung

### problem

Just want to start saying that i and my company found Cloudstack to be a very interesting project and we are looking at it as a replacement for our oVirt env.
We are in the middle of a POC so this is our first cluster at the moment.

I am using S3 as secondary storage with a NFS staging connected to it.
My consoleproxy VM starts without issues but my SSVM refuse to starts and i see following error in my logs

```
2025-03-06 13:19:14,173 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-135:[ctx-9dbacb5f, job-1299/job-3426]) (logid:5e35be66) Unable to complete AsyncJobVO: {id:3426, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2
Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAES
gAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAABinQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 95037156360330, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Mar 06 13:19:09 CET 2025, re
moved: null}, job origin: 1299 java.lang.NullPointerException: Cannot invoke "String.split(String)" because "url" is null
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.addSecondaryStorageServerAddressToBuffer(SecondaryStorageManagerImpl.java:1228)
at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.finalizeVirtualMachineProfile(SecondaryStorageManagerImpl.java:1216)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1272)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5467)
at jdk.internal.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

```

My guess it has to do with that the URL under Infrastructure -> Secondary storage is null? (See picture)
![Image](https://github.com/user-attachments/assets/691c9f77-7fa7-494b-8047-c846ca8ff79d)

Even using CMK the url field is null or not even exist in the response.
```
(localcloud) 🐱 > list imagestores id=a8bbf162-cc05-4cac-92d2-f18d43627cdd filter=name,url
{
"count": 1,
"imagestore": [
{
"name": "S3"
}
]
}
```

Also if i try to enter the secondary storage and browse it i see following error log that might be connected to this.
```
2025-03-06 13:56:18,487 ERROR [c.c.a.ApiServer] (qtp652176954-348:[ctx-c959a22d, ctx-21cc0661]) (logid:6136e9bd) unhandled exception executing api command: [Ljava.lang.String;@2f1ab954 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listImageStoreObjects(StorageBrowserImpl.java:128)
at org.apache.cloudstack.api.command.admin.storage.ListImageStoreObjectsCmd.execute(ListImageStoreObjectsCmd.java:72)

```

### versions

ACS: 4.20.0
Hypervisor: KVM with Rocky 9.5
Network: OpenVswitch 3.3
Primary Storage: Ceph RBD
Secondary Storage: NetApp S3 + Ceph NFS Staging

### The steps to reproduce the bug

1. Add a S3 Secondary storage either using WebUI or CMK/API
2. Wait for templates to download and SystemVMs to start
3. ConsoleProxy starts while SSVM will not start.

### What to do about it?
Is S3 as Secondary storage stable enough to use in a prod env?

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie in SecondaryStorageManagerImpl.java bei addSecondaryStorageServerAddressToBuffer (Zeile 1228) und finalizeVirtualMachineProfile (Zeile 1216), und untersuchen Sie anschließend StorageBrowserImpl.java bei listObjectsInStore (Zeile 183). Reproduzieren Sie das Problem mit einem S3-Sekundärspeicher und der aufgeführten Konfiguration, und überprüfen Sie dabei die Antwort der image-store API sowie den Start der SSVM; die Aufgabe ist abgeschlossen, wenn die SSVM ohne den Fehler wegen der null URL startet und beim Durchsuchen des Speichers kein fehlender Remote-Endpunkt mehr gemeldet wird.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
backend, cloud
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
52/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.