CKS: scale-down with separate etcd revokes wrong SSH rules and fails (4.22.1.1)

### problem

On CloudStack 4.22.1.1, a fresh CKS cluster with three separate etcd nodes can be created and scaled from one to two workers successfully, but scaling back to one worker fails with API error 530 while updating SSH network rules.

The backend has already removed the worker when it fails. It revokes the first etcd SSH firewall rule (50000), leaves the control/worker firewall range (2222–2224) active, and then conflicts when creating the replacement range (2222–2223). It also removes the control/worker SSH forwards and the first etcd SSH forward. The cluster becomes Alert with a stale worker count.

This controlled reproduction uses **three sequential native API mutations on one fresh cluster**, with Kubernetes **1.33.13 throughout**. Creation and scale-up were allowed to finish and pass node/pod/API readiness checks before proceeding. There was no Kubernetes upgrade, concurrent lifecycle operation, UI involvement, autoscaling, manual firewall customization, or per-node offering override.

### versions

- CloudStack management server and KVM agent: **4.22.1.1**. Installed package verification reports only local configuration differences; backend binaries are unchanged.
- Advanced zone, isolated network, CloudStack virtual router, shared NFS primary storage.
- SystemVM Template (KVM), Debian 12 guest OS; uniform offering: 2 vCPU, 4 GiB RAM, fixed 80 GiB root disk for all node types.
- Kubernetes 1.33.13, Calico 3.31.5. The ISO is custom-built with official component releases, includes the native etcd 3.5.24 binaries needed for separate-etcd bootstrap, and includes Headlamp and an upgrade-readiness hook. No upgrade was performed in this reproduction.
- `cloud.kubernetes.etcd.node.start.port=50000` (default).
- One control node, three separate etcd VMs, initially one worker. Normal tenant API credentials.

### The steps to reproduce the bug

The following are the complete application parameters for the three mutations, submitted via HTTP POST to `/client/api`. Standard `response=json` and API-key/signature authentication are omitted here. UUIDs identify this lab's zone, registered 1.33.13 ISO, offering and cluster; substitute equivalent IDs in another environment. No existing network was supplied: CloudStack created the isolated network and its rules automatically.

```json
{
"command": "createKubernetesCluster",
"name": "cks-etcd-minimal-20260912",
"zoneid": "5ebb765f-3f58-4574-acde-4df589dd478e",
"kubernetesversionid": "759912e1-1fd1-48c9-9d1c-da3de44834cc",
"serviceofferingid": "db6694af-7532-414d-b5f2-0300d95656f4",
"noderootdisksize": 80,
"clustertype": "CloudManaged",
"hypervisor": "kvm",
"controlnodes": 1,
"etcdnodes": 3,
"size": 1
}
```

```json
{
"command": "scaleKubernetesCluster",
"id": "3b74274b-6554-40c2-b2cf-4329d4308544",
"size": 2
}
```

```json
{
"command": "scaleKubernetesCluster",
"id": "3b74274b-6554-40c2-b2cf-4329d4308544",
"size": 1
}
```

Wait for each async job before proceeding:

| Operation | Job ID | Result / completion (UTC−03) |
|---|---|---|
| Create, size=1 | `821341c9-4fdd-45a5-ab0c-7838790880fb` | jobstatus=1, 2026-09-12 00:27:59; health passed |
| Scale, size=2 | `927ddb07-b482-4b16-a568-926249c79bec` | jobstatus=1, 00:29:52; health passed |
| Scale, size=1 | `b860f7d3-cd52-4396-b9ce-b3ee03eb4d5d` | jobstatus=2, error 530, 00:30:08 |

`queryAsyncJobResult(jobid=...)` was polled every five seconds. Read-only `listKubernetesClusters`, `getKubernetesClusterConfig`, `listPublicIpAddresses`, `listFirewallRules` and `listPortForwardingRules` calls captured state between steps; kubectl checked nodes, pods and `/readyz`. No fourth cluster mutation was submitted after the failure. The cluster remains available for investigation.

#### Expected result

Scale-down succeeds, the cluster reports one worker and Running, and SSH forwarding for the remaining control/worker and all three etcd VMs is preserved.

#### Actual results and relevant logs

The complete nested error states that 2222–2223 conflicts with the still-active 2222–2224 firewall rule. These are verbatim relevant management-server log lines from the failed job (unrelated lines omitted):

```text
2026-09-12 00:29:57,354 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) Removing vm VM instance {"id":519,"instanceName":"i-39-519-VM","state":"Running","type":"User","uuid":"939ddd32-c36f-461b-8771-c3a55d51647f"} from cluster KubernetesCluster {"id":79,"name":"cks-etcd-minimal-20260912","uuid":"3b74274b-6554-40c2-b2cf-4329d4308544"}
2026-09-12 00:30:07,904 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The SSH firewall rule null with the id 1600 was revoked
2026-09-12 00:30:07,905 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The Port forwarding rule [null] with the id [1601] was mark as revoked.
2026-09-12 00:30:07,907 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The Port forwarding rule [null] with the id [1607] was mark as revoked.
2026-09-12 00:30:07,907 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The Port forwarding rule [null] with the id [1608] was mark as revoked.
2026-09-12 00:30:08,761 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f, ctx-0f06110e]) (logid:b860f7d3) Rules ([FirewallRule {"id":1606,"networkId":281,"purpose":"Firewall","state":"Active","uuid":"9439cbc3-faeb-4b6a-b7a6-66fdd167343b"}] and [FirewallRule {"id":1610,"networkId":281,"purpose":"Firewall","state":"Staged","uuid":"d2e0ba0c-931d-45a8-a093-770dc73e636e"}]) have conflicting port ranges.
2026-09-12 00:30:08,768 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) Scaling failed for Kubernetes cluster KubernetesCluster {"id":79,"name":"cks-etcd-minimal-20260912","uuid":"3b74274b-6554-40c2-b2cf-4329d4308544"}, unable to update network rules com.cloud.exception.ManagementServerException: Failed to provision firewall rules for SSH access for the Kubernetes cluster : cks-etcd-minimal-20260912
Caused by: com.cloud.exception.NetworkRuleConflictException: The range specified, 2222-2223, conflicts with rule FirewallRule {"id":1606,"networkId":281,"purpose":"Firewall","state":"Active","uuid":"9439cbc3-faeb-4b6a-b7a6-66fdd167343b"} which has 2222-2224
at com.cloud.network.firewall.FirewallManagerImpl.detectRulesConflict(FirewallManagerImpl.java:486)
at com.cloud.network.firewall.FirewallManagerImpl.lambda$createFirewallRule$0(FirewallManagerImpl.java:265)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.network.firewall.FirewallManagerImpl.createFirewallRule(FirewallManagerImpl.java:255)
```

Rule identities reconstructed from the rule-application commands and API snapshots:

| Rule | Purpose / ports | Observed change |
|---|---|---|
| 1595 | Initial control/worker firewall, 2222–2223 | Correctly revoked during scale-up |
| 1600 | First etcd firewall, 50000 | Incorrectly revoked during scale-down |
| 1602 / 1604 | Other etcd firewalls, 50001 / 50002 | Remain active |
| 1606 | Replacement control/worker firewall, 2222–2224 | Created during scale-up; incorrectly left active during scale-down |
| 1601 | First etcd port forward, 50000→22 | Removed during scale-down |
| 1607 / 1608 | Control / retained-worker port forwards, 2222→22 / 2223→22 | Removed during scale-down |
| 1610 | Attempted replacement firewall, 2222–2223 | Rejected due to overlap with 1606 |

After failure, `listPortForwardingRules` returns only 50001→22 and 50002→22. `listFirewallRules` returns 2222–2224, 6443, 50001 and 50002. CloudStack reports Alert and `size=2`, while only one worker VM remains. The five remaining VMs are Running. kubectl shows the control and one worker Ready on **v1.33.13**, and `/readyz` still returns `ok`.

### What to do about it?

In [4.22.1.1 `removeSshFirewallRule`](https://github.com/apache/cloudstack/blob/4.22.1.1/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java#L546-L558), the selector accepts either a firewall starting at 2222 **or any corresponding port forward whose destination starts at 22**, then breaks after the first match:

```java
if (Objects.equals(firewallRule.getSourcePortStart(), CLUSTER_NODES_DEFAULT_START_SSH_PORT)
|| (Objects.nonNull(pfRule) && pfRule.getDestinationPortStart() == DEFAULT_SSH_PORT)) {
rule = firewallRule;
firewallService.revokeIngressFwRule(firewallRule.getId(), true);
break;
}
```

The first etcd SSH rule also satisfies that condition. The preceding scale-up recreated the control/worker rule after the etcd rules; the next scale selected etcd rule 1600 first.

[`scaleKubernetesClusterIsolatedNetworkRules`](https://github.com/apache/cloudstack/blob/4.22.1.1/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java#L134-L155) then uses the selected rule's end port as the upper bound for port-forward cleanup, yielding **2222–50000**. This explains the removal of both regular-node SSH forwards and the first etcd forward. Recreating 2222–2223 then conflicts with the untouched general rule.

Please distinguish control/worker SSH rules from separate-etcd rules during selection and bound the forwarding cleanup accordingly. A regression test covering fresh three-etcd creation → worker scale 1→2→1, including rule ordering after recreation, should catch this. The partial VM removal and stale cluster count on a network-rule failure also merit recovery handling.

Bắt đầu trong KubernetesClusterResourceModifierActionWorker.removeSshFirewallRule và KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules; theo dõi cách các quy tắc firewall và port-forward được chọn và giới hạn. Tái hiện chuỗi được ghi trong tài liệu gồm tạo, scale từ 1→2 và scale từ 2→1, sau đó dùng một regression test để kiểm thử thứ tự của các quy tắc được tạo lại. Được xem là hoàn tất khi scale-down thành công, duy trì quyền truy cập SSH riêng biệt-etcd và quyền truy cập SSH control/worker còn lại, đồng thời giữ trạng thái cluster nhất quán.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.