CloudStack workload cluster created by the CLI rolls out too many ETCD nodes on upgrade from latest minor release (0.16.5 to 0.17.x) with the API.
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
Workload cluster that was created by the CLI with release 0.16.5 fails on upgrade to the latest release 0.17.0 (seems to exist when upgrade to any 0.17.x version) when using `kubectl apply`.
Error observed in e2e test logs.
```
cluster.go:2049: failed to validate cluster [CAPI cluster main-i-0647d-924ddf8-w-0 not ready yet. MaxNumberOfEtcdMachinesReached, failed to validate controlplane node ready: validating node version: kubernetes version v1.23.17-eks-bbbebb8 does not match expected version 1.24]
```
What happens is that it spins up some new etcd nodes (3 is the max currently) and never scales back down to the expected number 1. It looks like etcd reached the 2X replicas (the MaxNumberOfEtcdMachinesReached ), which stopped the cluster from doing anything further with etcd and stopping it from upgrading successfully causing k8s version to not match
The old control plane node is not deleted and the kubeapi-server was not contactable (this does not always happen).
From further investigation, it seems like there's a race condition somewhere that results in etcd rolling out before machine templates are ready (and the new rolled out CloudStack machines reference the old template for kubernetes version 1.23). So far it seems to only happen in CloudStack.
```
2023-09-06T21:59:18Z ERROR Reconciler error {"controller": "etcdadmcluster", "controllerGroup": "etcdcluster.cluster.x-k8s.io", "controllerKind": "EtcdadmCluster", "EtcdadmCluster": {"name":"eksa-test-924ddf8-w-0-etcd","namespace":"eksa-system"}, "namespace": "eksa-system", "name": "eksa-test-924ddf8-w-0-etcd", "reconcileID": "d01654ed-9410-4f9f-a11b-86797e40dc4a", "error": "failed to retrieve CloudStackMachineTemplate external object \"eksa-system\"/\"eksa-test-924ddf8-w-0-etcd-template-1694035722736\": cloudstackmachinetemplates.infrastructure.cluster.x-k8s.io \"eksa-test-924ddf8-w-0-etcd-template-1694035722736\" not found", "errorVerbose": "cloudstackmachinetemplates.infrastructure.cluster.x-k8s.io \"eksa-test-924ddf8-w-0-etcd-template-1694035722736\" not found\nfailed to retrieve CloudStackMachineTemplate external object \"eksa-system\"/\"eksa-test-924ddf8-w-0-etcd-template-1694035722736\"\nsigs.k8s.io/cluster-api/controllers/external.Get\n\tsigs.k8s.io/cluster-api@v1.3.3/controllers/external/util.go:44\ngithub.com/aws/etcdadm-controller/controllers.(*EtcdadmClusterReconciler).reconcileExternalReference\n\tgithub.com/aws/etcdadm-controller/controllers/helpers.go:230\ngithub.com/aws/etcdadm-controller/controllers.(*EtcdadmClusterReconciler).reconcile\n\tgithub.com/aws/etcdadm-controller/controllers/controller.go:200\ngithub.com/aws/etcdadm-controller/controllers.(*EtcdadmClusterReconciler).Reconcile\n\tgithub.com/aws/etcdadm-controller/controllers/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
...
2023-09-06T22:01:59Z INFO controllers.EtcdadmCluster.eksa-test-924ddf8-w-0-etcd Etcd cluster needs a rollout {"totalMachines": 3, "needRollout": 1}
2023-09-06T22:04:04Z INFO controllers.EtcdadmCluster.eksa-test-924ddf8-w-0-etcd Cluster has reached the max number of machines, won't create new machines until at least one is deleted {"totalMachines": 3}
...
2023-09-06T22:05:02Z INFO controllers.EtcdadmCluster.eksa-test-924ddf8-w-0-etcd Controlplane upgrade has completed, deleting older outdated etcd members: [eksa-test-924ddf8-w-0-etcd-6vts8]
```
After manually cleaning up the first new etcd machine that got rolled out
```
2023-09-06T22:05:19Z INFO controllers.EtcdadmCluster.eksa-test-924ddf8-w-0-etcd Outdated etcd members deleted, removing controlplane-upgrade complete annotation
2023-09-06T22:05:19Z INFO controllers.EtcdadmCluster.eksa-test-924ddf8-w-0-etcd Outdated etcd members deleted, setting EtcdClusterHasNoOutdatedMachines to true
```
This only seems to happen when the workload cluster is configured with only 1 Etcd node.
Further investigation reveals that the controller reconciles a cluster spec where the Cluster.Spec.KubernetesVersion is 1.24 and the `CloudStackMachineConfig.Spec.Template.Name` still references the old 1.23 template:
For vSphere, when you upgrade the k8s version, you also need to update the template which has the proper EKS-D tags. So the controller won’t start reconciliation until it sees that the tags on VSphereMachineConfig.spec.template matches `Cluster.Spec.KubernetesVersion`. We don’t have a similar check on CloudStack and adding one seems to fix the issue.
**What you expected to happen**:
Workload cluster machines are all upgraded to kubernetes 1.24 and use the latest bundle and templates.
**How to reproduce it (as minimally and precisely as possible)**:
Use these steps to reproduce. Each step indicate whether CLI or API was used.
Cluster configurations: 1 Control Plane, 1 ETCD, 1 Worker Node
- CLI: Created a management with kubernetes version 1.23 using the latest minor release 0.16.5
- CLI: Create a workload cluster with kubernetes version 1.23 using the latest minor release 0.16.5
- Upgrade the management cluster to kubernetes 1.24 using release 0.17.0.
- API: Upgrade the workload cluster to kubernetes 1.24 using the new EksaVersion referenced.
**Anything else we need to know?**:
**Environment**:
- EKS Anywhere Release: 0.16.5 -> 0.17.0
- EKS Distro Release: v1-23-eks-26 - > v1-24-eks-23
Contributor guide
Research direction
Start by tracing the EtcdadmClusterReconciler reconciliation around CloudStackMachineConfig.Spec.Template.Name and the CloudStackMachineTemplate external reference, comparing it with the VSphereMachineConfig version check described in the issue. Reproduce the 0.16.5-to-0.17.0 upgrade with one ETCD node and inspect the e2e logs. Done means CloudStack waits for the new Kubernetes version template before rolling out ETCD machines and the upgrade completes without exceeding the expected node count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100