apache / apache/cloudstack

CKS: Scaling of Kubernetes Cluster fails if there is external worker node is added

Open
#12,633 0 comments 0 reactions 2 assignees Claimed by @nvazquez View on GitHub
component:cks
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

CKS: Scaling of Kubernetes Cluster fails if there is external worker node is added

### versions

ACS 4.22 , 4.22.1 -nightly

### The steps to reproduce the bug

1. Create a cks cluster with size 1 for worker nodes

2. Add an external node to the cks cluster

https://docs.cloudstack.apache.org/en/4.22.0.0/plugins/cloudstack-kubernetes-service.html#add-an-external-vm-instance-as-a-worker-node-to-a-kubernetes-cluster

3. Scale the cks cluster to size 3 ( worker nodes)

4. Cks cluster scaling fails and cluster goes into alert state

Image

logs

```
2026-02-12 09:59:04,050 DEBUG [c.c.u.d.T.Transaction] (API-Job-Executor-46:[ctx-efb09e57, job-241, ctx-3d8bf3ff, ctx-7b1fe543]) (logid:76986184) Rolling back the transaction: Time = 7 Name = API-Job-Executor-46; called by -TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-Transaction.execute:36-FirewallManagerImpl.createFirewallRule:255-FirewallManagerImpl.createIngressFirewallRule:207-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:77-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:569-AopUtils.invokeJoinpointUsingReflection:344-ReflectiveMethodInvocation.invokeJoinpoint:198
2026-02-12 09:59:04,063 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-46:[ctx-efb09e57, job-241, ctx-3d8bf3ff]) (logid:76986184) Scaling failed for Kubernetes cluster : test-cks, unable to update network rules com.cloud.exception.ManagementServerException: Failed to provision firewall rules for SSH access for the Kubernetes cluster : test-cks
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.addFirewallRulesForNodes(KubernetesClusterActionWorker.java:1037)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.createFirewallRules(KubernetesClusterResourceModifierActionWorker.java:668)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.setupKubernetesClusterIsolatedNetworkRules(KubernetesClusterResourceModifierActionWorker.java:699)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules(KubernetesClusterScaleWorker.java:151)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:192)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:473)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:509)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:584)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:2142)
at jdk.internal.reflect.GeneratedMethodAccessor721.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy537.scaleKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKu

```

### What to do about it?

CKS cluster scaling should succeed if there is an external node added to the cluster

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.