CKS upgrade fails in the following scenario
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 19h
- Merged PRs (30d)
- 32
Description
### problem
CKS upgrade fails in the following scenario
### versions
ACS 4.22,423
### The steps to reproduce the bug
1. Register 2 cks iso ( 1.36 and 1.37)
2. Launch a cks cluster with 1.36
```
create kubernetescluster name=cks-npe zoneid=157cb8ca-1c40-4a1f-a737-8bddb1dcb2fd kubernetesversionid=e8d2b996-54c8-457a-9a54-985ebc46b332 networkid=f2d97bea-3e96-44bd-ae16-fd98d7ea8a8e controlnodes=1 size=2 nodeofferings[0].node=control nodeofferings[0].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 nodeofferings[1].node=worker nodeofferings[1].offering=d7892a54-8604-44ab-8892-dcd1357571a0 hypervisor=KVM
```
3. Upgrade the cks cluster to 1.37
```
(localcloud) 🐱 > upgrade kubernetescluster id=8912cd8d-29bd-44de-b5fc-6965e607cc2f kubernetesversionid=ac8576c2-8ddd-4008-b604-ec804220578e
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd",
"completed": "2026-09-10T18:45:53+0000",
"created": "2026-09-10T18:45:53+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "7067c3b9-0510-44dd-bf8e-903ac61ae330",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Invalid service offering associated with Kubernetes cluster : cks-npe"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
```
Logs
```
🙈 Error: async API failed for job 7067c3b9-0510-44dd-bf8e-903ac61ae330
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:7067c3b9"
2026-09-10 18:45:53,428 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T19:00:53Z\",\"sessionkey\":\"oMUOvbLNNYCXMWQFm7q2WsLWymA\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"8912cd8d-29bd-44de-b5fc-6965e607cc2f\",\"cmdEventType\":\"KUBERNETES.CLUSTER.UPGRADE\",\"kubernetesversionid\":\"ac8576c2-8ddd-4008-b604-ec804220578e\",\"signatureversion\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"10164\",\"id\":\"8912cd8d-29bd-44de-b5fc-6965e607cc2f\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.version.KubernetesSupportedVersion\\\":\\\"ac8576c2-8ddd-4008-b604-ec804220578e\\\",\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"8912cd8d-29bd-44de-b5fc-6965e607cc2f\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1747,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"7067c3b9-0510-44dd-bf8e-903ac61ae330"}
2026-09-10 18:45:53,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Complete async job-1747, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Invalid service offering associated with Kubernetes cluster : cks-npe"}
2026-09-10 18:45:53,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Publish async job-1747 complete on message bus
2026-09-10 18:45:53,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Wake up jobs related to job-1747
2026-09-10 18:45:53,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Update db status for job-1747
2026-09-10 18:45:53,457 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Wake up jobs joined with job-1747 and disjoin all subjobs created from job- 1747
2026-09-10 18:45:53,466 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd for job-1747
2026-09-10 18:45:53,466 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-59:[ctx-4251e64e, job-1747]) (logid:7067c3b9) Remove job-1747 from job monitoring
````
Could you be related to https://github.com/apache/cloudstack/issues/14122
### What to do about it?
Upgrade of the cks cluster should succeed
Contributor guide
Research direction
Start with the UpgradeKubernetesClusterCmd entry point and review the linked pull request #14138, then reproduce the upgrade using the documented create and upgrade commands. Trace the invalid service offering error shown in the management-server log; done means upgrading the 1.36 CKS cluster to 1.37 succeeds without error 530.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- backend-api-design, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100