Scaling issues with cks cluster
- Langage dominant
- Java
- Étoiles
- 3.1k
- Forks
- 1.4k
- Merge moyen
- 6 j 19 h
- PR mergées (30 j)
- 32
Description
### problem
Scaling issues with cks cluster
### versions
ACS 4.23, 4.22
Hypervisor : KVM
CKS : 1.36
### The steps to reproduce the bug
Steps to reproduce the issue
1. Create 2 service offerings
2. create kubernetescluster name=cks-npe zoneid=157cb8ca-1c40-4a1f-a737-8bddb1dcb2fd kubernetesversionid=e8d2b996-54c8-457a-9a54-985ebc46b332 networkid=f2d97bea-3e96-44bd-ae16-fd98d7ea8a8e controlnodes=1 size=2 nodeofferings[0].node=control nodeofferings[0].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 nodeofferings[1].node=worker nodeofferings[1].offering=d7892a54-8604-44ab-8892-dcd1357571a0 hypervisor=KVM
Check the database
```
mysql> SELECT name, state, service_offering_id, control_node_service_offering_id, worker_node_service_offering_id, etcd_node_service_offering_id, etcd_node_count FROM cloud.kubernetes_cluster WHERE name='cks-npe' AND removed IS NULL;
+---------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| name | state | service_offering_id | control_node_service_offering_id | worker_node_service_offering_id | etcd_node_service_offering_id | etcd_node_count |
+---------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| cks-npe | Running | NULL | 26 | 27 | NULL | NULL |
+---------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
1 row in set (0.00 sec)
```
3. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0
exception
```
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-09-10T10:41:08+0000",
"created": "2026-09-10T10:41:08+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "034171ef-25d7-4243-bbd7-6524431309af",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Cannot invoke \"com.cloud.offering.ServiceOffering.getRamSize()\" because \"existingServiceOffering\" is null"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
```
Logs
```
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:034171ef"
2026-09-10 10:41:08,704 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T10:56:08Z\",\"sessionkey\":\"v1xKdYPWziwzZfJE-1T4fxIgyyI\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"serviceofferingid\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9114\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\",\\\"interface com.cloud.offering.ServiceOffering\\\":\\\"d7892a54-8604-44ab-8892-dcd1357571a0\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1430,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"034171ef-25d7-4243-bbd7-6524431309af"}
2026-09-10 10:41:08,706 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430, ctx-e097f2dc]) (logid:034171ef) Account [Account [{"accountName":"admin","id":2,"uuid":"47dc9e4c-83f6-11f1-aed8-bc24112069f3"}]] has access to resource.
2026-09-10 10:41:08,735 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Complete async job-1430, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null"}
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Publish async job-1430 complete on message bus
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Wake up jobs related to job-1430
2026-09-10 10:41:08,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Update db status for job-1430
2026-09-10 10:41:08,738 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Wake up jobs joined with job-1430 and disjoin all subjobs created from job- 1430
2026-09-10 10:41:08,751 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1430
2026-09-10 10:41:08,751 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-22:[ctx-606a2296, job-1430]) (logid:034171ef) Remove job-1430 from job monitoring
```
4. Scale the cks cluster by providing only the size parameter
```
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-09-10T10:57:23+0000",
"created": "2026-09-10T10:57:23+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "0f928699-0ce2-47de-b9db-eeb8401c406d",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "The Kubernetes cluster cks-npe does not have a service offering set for node type ETCD"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 0f928699-0ce2-47de-b9db-eeb8401c406d
```
Logs
```
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:0f928699"
2026-09-10 10:57:23,386 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:12:23Z\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9118\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1431,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"0f928699-0ce2-47de-b9db-eeb8401c406d"}
2026-09-10 10:57:23,410 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-23:[ctx-3a32d20f, job-1431, ctx-eca8a9bb]) (logid:0f928699) Scaling Kubernetes cluster KubernetesCluster {"id":7,"name":"cks-npe","uuid":"7944a42f-fcf2-4f4e-932a-e7b59e0724b1"}
2026-09-10 10:57:23,412 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-23:[ctx-3a32d20f, job-1431, ctx-eca8a9bb]) (logid:0f928699) The Kubernetes cluster cks-npe does not have a service offering set for node type ETCD
2026-09-10 10:57:23,412 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-23:[ctx-3a32d20f, job-1431, ctx-eca8a9bb]) (logid:0f928699) Revoking launch permission for systemVM Template
2026-09-10 10:57:23,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Complete async job-1431, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"The Kubernetes cluster cks-npe does not have a service offering set for node type ETCD"}
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Publish async job-1431 complete on message bus
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Wake up jobs related to job-1431
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Update db status for job-1431
2026-09-10 10:57:23,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Wake up jobs joined with job-1431 and disjoin all subjobs created from job- 1431
2026-09-10 10:57:23,445 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1431
2026-09-10 10:57:23,445 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-23:[ctx-3a32d20f, job-1431]) (logid:0f928699) Remove job-1431 from job monitoring
```
5. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 nodeofferings[0].node=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
```
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 size=3 nodeofferings[0].node=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-09-10T11:04:52+0000",
"created": "2026-09-10T11:04:51+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "8dabeecf-0c6a-4771-b2d9-502865ffd944",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Cannot invoke \"com.cloud.offering.ServiceOffering.getRamSize()\" because \"existingServiceOffering\" is null"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 8dabeecf-0c6a-4771-b2d9-502865ffd944
```
logs
```
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:8dabeecf"
2026-09-10 11:04:51,999 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:19:51Z\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"nodeofferings[0].offering\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"nodeofferings[0].node\":\"etcd\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9121\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1432,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8dabeecf-0c6a-4771-b2d9-502865ffd944"}
2026-09-10 11:04:52,015 DEBUG [c.c.k.c.KubernetesServiceHelperImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432, ctx-75ed24fb]) (logid:8dabeecf) Node Type: 'etcd' should use Service Offering ID: 'd7892a54-8604-44ab-8892-dcd1357571a0'
2026-09-10 11:04:52,027 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Complete async job-1432, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null"}
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Publish async job-1432 complete on message bus
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Wake up jobs related to job-1432
2026-09-10 11:04:52,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Update db status for job-1432
2026-09-10 11:04:52,029 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Wake up jobs joined with job-1432 and disjoin all subjobs created from job- 1432
2026-09-10 11:04:52,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1432
2026-09-10 11:04:52,044 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-24:[ctx-ed8d2ed5, job-1432]) (logid:8dabeecf) Remove job-1432 from job monitoring
```
6. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 nodeofferings[0].key=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
```
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 nodeofferings[0].key=etcd nodeofferings[0].offering=d7892a54-8604-44ab-8892-dcd1357571a0
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-09-10T11:07:43+0000",
"created": "2026-09-10T11:07:43+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "051ee202-e648-438e-bd19-f1aa885044f4",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Incomplete Node Type to Service Offering ID mapping: 'null' -> 'd7892a54-8604-44ab-8892-dcd1357571a0'"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 051ee202-e648-438e-bd19-f1aa885044f4
```
logs
```
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:051ee202"
2026-09-10 11:07:43,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:22:43Z\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"nodeofferings[0].key\":\"etcd\",\"nodeofferings[0].offering\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9124\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1433,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"051ee202-e648-438e-bd19-f1aa885044f4"}
2026-09-10 11:07:43,564 ERROR [c.c.k.c.KubernetesServiceHelperImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433, ctx-6bdd9a94]) (logid:051ee202) Incomplete Node Type to Service Offering ID mapping: 'null' -> 'd7892a54-8604-44ab-8892-dcd1357571a0'
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Complete async job-1433, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Incomplete Node Type to Service Offering ID mapping: 'null' -> 'd7892a54-8604-44ab-8892-dcd1357571a0'"}
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Publish async job-1433 complete on message bus
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Wake up jobs related to job-1433
2026-09-10 11:07:43,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Update db status for job-1433
2026-09-10 11:07:43,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Wake up jobs joined with job-1433 and disjoin all subjobs created from job- 1433
2026-09-10 11:07:43,585 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1433
2026-09-10 11:07:43,585 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-25:[ctx-0ec6aa90, job-1433]) (logid:051ee202) Remove job-1433 from job monitoring
```
7. scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 autoscalingenabled=true minsize=3 maxsize=5 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0
```
(localcloud) 🐱 > scale kubernetescluster id=7944a42f-fcf2-4f4e-932a-e7b59e0724b1 autoscalingenabled=true minsize=3 maxsize=5 serviceofferingid=d7892a54-8604-44ab-8892-dcd1357571a0
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-09-10T11:12:07+0000",
"created": "2026-09-10T11:12:07+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "1f5ab1ce-0cd6-43ff-b0c7-9e87c19507c3",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Cannot invoke \"com.cloud.offering.ServiceOffering.getRamSize()\" because \"existingServiceOffering\" is null"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 1f5ab1ce-0cd6-43ff-b0c7-9e87c19507c3
```
```
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:1f5ab1ce"
2026-09-10 11:12:07,190 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T11:27:07Z\",\"autoscalingenabled\":\"true\",\"sessionkey\":\"CaP7TfxTrU4jexSnhOGcatQMFt8\",\"httpmethod\":\"POST\",\"minsize\":\"3\",\"ctxAccountId\":\"2\",\"uuid\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"serviceofferingid\":\"d7892a54-8604-44ab-8892-dcd1357571a0\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9127\",\"id\":\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\",\"maxsize\":\"5\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"7944a42f-fcf2-4f4e-932a-e7b59e0724b1\\\",\\\"interface com.cloud.offering.ServiceOffering\\\":\\\"d7892a54-8604-44ab-8892-dcd1357571a0\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1434,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"1f5ab1ce-0cd6-43ff-b0c7-9e87c19507c3"}
2026-09-10 11:12:07,197 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434, ctx-17662367]) (logid:1f5ab1ce) Account [Account [{"accountName":"admin","id":2,"uuid":"47dc9e4c-83f6-11f1-aed8-bc24112069f3"}]] has access to resource.
2026-09-10 11:12:07,225 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.NullPointerException: Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null
2026-09-10 11:12:07,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Complete async job-1434, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Cannot invoke "com.cloud.offering.ServiceOffering.getRamSize()" because "existingServiceOffering" is null"}
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Publish async job-1434 complete on message bus
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Wake up jobs related to job-1434
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Update db status for job-1434
2026-09-10 11:12:07,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Wake up jobs joined with job-1434 and disjoin all subjobs created from job- 1434
2026-09-10 11:12:07,250 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1434
2026-09-10 11:12:07,250 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-26:[ctx-39b4db84, job-1434]) (logid:1f5ab1ce) Remove job-1434 from job monitoring
```
8. Create a cks cluster with etcd node
```
create kubernetescluster name=cks-etcd3 \
zoneid=157cb8ca-1c40-4a1f-a737-8bddb1dcb2fd \
kubernetesversionid=e8d2b996-54c8-457a-9a54-985ebc46b332 \
networkid=f2d97bea-3e96-44bd-ae16-fd98d7ea8a8e \
controlnodes=1 size=2 etcdnodes=1 \
nodeofferings[0].node=control nodeofferings[0].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 \
nodeofferings[1].node=worker nodeofferings[1].offering=d7892a54-8604-44ab-8892-dcd1357571a0 \
nodeofferings[2].node=etcd nodeofferings[2].offering=bf332776-e6c1-43cb-922d-7de6bb5d38a7 \
hypervisor=KVM
mysql> SELECT name, state, service_offering_id, control_node_service_offering_id, worker_node_service_offering_id, etcd_node_service_offering_id, etcd_node_count FROM cloud.kubernetes_cluster WHERE name='cks-etcd3' AND removed IS NULL;
+-----------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| name | state | service_offering_id | control_node_service_offering_id | worker_node_service_offering_id | etcd_node_service_offering_id | etcd_node_count |
+-----------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
| cks-etcd3 | Running | NULL | 26 | 27 | 26 | 1 |
+-----------+---------+---------------------+----------------------------------+---------------------------------+-------------------------------+-----------------+
1 row in set (0.00 sec)
```
```
(localcloud) 🐱 > scale kubernetescluster id=bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da size=3
{
"account": "admin",
"accountid": "47dc9e4c-83f6-11f1-aed8-bc24112069f3",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-09-10T11:45:41+0000",
"created": "2026-09-10T11:45:41+0000",
"domainid": "c92b36a9-83f5-11f1-aed8-bc24112069f3",
"domainpath": "ROOT",
"jobid": "3ac3beef-852d-46a7-be6b-04c956006e9d",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Scaling failed for Kubernetes cluster : cks-etcd3, cluster service offering not found"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "47dcda68-83f6-11f1-aed8-bc24112069f3"
}
🙈 Error: async API failed for job 3ac3beef-852d-46a7-be6b-04c956006e9d
```
```
root@Cloudstack-423:/home/ubuntu# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:3ac3beef"
2026-09-10 11:45:41,445 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd","cmdInfo":"{\"expires\":\"2026-09-10T12:00:41Z\",\"sessionkey\":\"-6T60SNJxljbmS-Daml5a3Wkwmo\",\"httpmethod\":\"POST\",\"ctxAccountId\":\"2\",\"uuid\":\"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da\",\"cmdEventType\":\"KUBERNETES.CLUSTER.SCALE\",\"signatureversion\":\"3\",\"size\":\"3\",\"response\":\"json\",\"ctxUserId\":\"2\",\"ctxStartEventId\":\"9271\",\"id\":\"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da\",\"ctxDetails\":\"{\\\"interface com.cloud.kubernetes.cluster.KubernetesCluster\\\":\\\"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da\\\"}\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":1464,"initMsid":206863092181491,"instanceId":null,"instanceType":"KubernetesCluster","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"3ac3beef-852d-46a7-be6b-04c956006e9d"}
2026-09-10 11:45:41,465 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-31:[ctx-f1091fa2, job-1464, ctx-39991b87]) (logid:3ac3beef) Scaling Kubernetes cluster KubernetesCluster {"id":9,"name":"cks-etcd3","uuid":"bfdaa852-cdfc-42f1-a7bb-1c50ff9a59da"}
2026-09-10 11:45:41,469 WARN [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-31:[ctx-f1091fa2, job-1464, ctx-39991b87]) (logid:3ac3beef) Scaling failed for Kubernetes cluster : cks-etcd3, cluster service offering not found
2026-09-10 11:45:41,469 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-31:[ctx-f1091fa2, job-1464, ctx-39991b87]) (logid:3ac3beef) Revoking launch permission for systemVM Template
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Complete async job-1464, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Scaling failed for Kubernetes cluster : cks-etcd3, cluster service offering not found"}
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Publish async job-1464 complete on message bus
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Wake up jobs related to job-1464
2026-09-10 11:45:41,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Update db status for job-1464
2026-09-10 11:45:41,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Wake up jobs joined with job-1464 and disjoin all subjobs created from job- 1464
2026-09-10 11:45:41,507 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Done executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd for job-1464
2026-09-10 11:45:41,507 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-31:[ctx-f1091fa2, job-1464]) (logid:3ac3beef) Remove job-1464 from job monitoring
```
### What to do about it?
Scaling should work without any exception message
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par ScaleKubernetesClusterCmd et KubernetesClusterScaleWorker, en utilisant les étapes de reproduction et les erreurs du management-server comme guide initial. Suivez la manière dont les offres de service sont sélectionnées pour les nœuds de contrôle, worker et ETCD. C'est terminé lorsque les demandes de mise à l'échelle du cluster présentées n'échouent plus à cause d'une offre de service null ou d'une offre ETCD manquante.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java, kubernetes
- Domaine
- backend, infrastructure
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 25/100