RollingUpgrades observed on workload clusters when upgrading management cluster
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
Occasionally, changes such as the one in https://github.com/aws/eks-anywhere/pull/3402 can cause minor changes in the CAPI resources which the EKS-A controller picks up. When a management cluster is upgraded and a new EKS-A controller is installed on a cluster, it begins applying these new resources when reconciling not just the management cluster, _but the workload clusters as well_. This causes _all_ of them to perform an unexpected rollingupgrade. I believe this rolling upgrade for workload clusters is an undesirable behavior for users and should be avoided or mitigated.
When I was running some large scale tests with many workload clusters, I observed that every single workload cluster began its rolling upgrade simultaneously. If I have ten workload clusters with VM's of 16GB and 8 CPU's, and all of them start a rolling upgrade simultaneously, that will require a huge spike in resources which I may not be prepared to handle. This issue becomes even more prevalent with larger VM's and more workload clusters.
**What you expected to happen**:
Ideally, management cluster upgrades would never trigger rolling upgrades on workload clusters as a side effect. However, given that they share some resources, these upgrades should at least be staggered, and the CLI should wait for them all to complete and for the workload clusters to stabilize. Workload clusters should be paused for the duration of the management cluster's upgrade and unpaused gradually, while waiting for the cluster to become ready.
**How to reproduce it (as minimally and precisely as possible)**:
Upgrade a mgmt cluster with a workload cluster from v0.11.3 to v0.12.x (or the code in main at the time of writing of this issue). The management cluster upgrade will trigger a rolling upgrade for the workload cluster.
**Subtasks**:
- [x] https://github.com/aws/eks-anywhere/issues/3543
- [ ] https://github.com/aws/eks-anywhere/issues/3544
**Environment**:
- EKS Anywhere Release:main
- EKS Distro Release:
Contributor guide
Research direction
Start by reproducing the management-cluster upgrade from v0.11.3 to v0.12.x and trace how the EKS-A controller reconciles CAPI resources for workload clusters. Review the completed subtask in issue 3543 and the remaining work in issue 3544. Done means workload upgrades are avoided or safely staggered, with the CLI waiting for workload clusters to stabilize.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100