apache / apache/cloudstack

CKS upgrade fails in the following scenario

Open
#14,148 1 comment 0 reactions 0 assignees Claimed by @Pearl1594 View on GitHub
bug component:kubernetes
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

CKS upgrade fails in the following scenario

### versions

ACS 4.22,423

### The steps to reproduce the bug

1. Register 2 cks iso ( 1.36 and 1.37)

2. Launch a cks cluster with 1.36

```
create kubernetescluster name=cks-npe zoneid=157cb8ca-1c40-4a1f-a737-8bddb1dcb2fd kubernetesversionid=e8d2b996-54c8-457a-9a54-985ebc46b332 networkid=f2d97bea-3e96-44bd-ae16-fd98d7ea8a8e controlnodes=1 size=2 nodeofferings[0].node=control nodeofferings[0].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 nodeofferings[1].node=worker nodeofferings[1].offering=d7892a54-8604-44ab-8892-dcd1357571a0 hypervisor=KVM

```
3. Upgrade the cks cluster to 1.37

```
(localcloud) 🐱 > upgrade kubernetescluster id=8912cd8d-29bd-44de-b5fc-6965e607cc2f kubernetesversionid=ac8576c2-8ddd-4008-b604-ec804220578e
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd",
"completed": "2026-09-10T18:45:53+0000",
"created": "2026-09-10T18:45:53+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "7067c3b9-0510-44dd-bf8e-903ac61ae330",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Invalid service offering associated with Kubernetes cluster : cks-npe"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}

```
Logs

```

🙈 Error: async API failed for job 7067c3b9-0510-44dd-bf8e-903ac61ae330


root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:7067c3b9"
2026-09-10 18:45:53,428 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T19:00:53Z\",\"sessionkey\":\"oMUOvbLNNYCXMWQFm7q2WsLWymA\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"8912cd8d-29bd-44de-b5fc-6965e607cc2f\",\"cmdEventType\":\"KUBERNETES.CLUSTER.UPGRADE\",\"kubernetesversionid\":\"ac8576c2-8ddd-4008-b604-ec804220578e\",\"signatureversion\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"10164\",\"id\":\"8912cd8d-29bd-44de-b5fc-6965e607cc2f\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.version.KubernetesSupportedVersion\\\":\\\"ac8576c2-8ddd-4008-b604-ec804220578e\\\",\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"8912cd8d-29bd-44de-b5fc-6965e607cc2f\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1747,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"7067c3b9-0510-44dd-bf8e-903ac61ae330"}
2026-09-10 18:45:53,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Complete async job-1747, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Invalid service offering associated with Kubernetes cluster : cks-npe"}
2026-09-10 18:45:53,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Publish async job-1747 complete on message bus
2026-09-10 18:45:53,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Wake up jobs related to job-1747
2026-09-10 18:45:53,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Update db status for job-1747
2026-09-10 18:45:53,457 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Wake up jobs joined with job-1747 and disjoin all subjobs created from job- 1747
2026-09-10 18:45:53,466 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd for job-1747
2026-09-10 18:45:53,466 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Remove job-1747 from job monitoring

````

Could you be related to https://github.com/apache/cloudstack/issues/14122

### What to do about it?

Upgrade of the cks cluster should succeed

Contributor guide

Open the contributing guide

Research direction

Start with the UpgradeKubernetesClusterCmd entry point and review the linked pull request #14138, then reproduce the upgrade using the documented create and upgrade commands. Trace the invalid service offering error shown in the management-server log; done means upgrading the 1.36 CKS cluster to 1.37 succeeds without error 530.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend-api-design, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.