aws / aws/eks-anywhere

RollingUpgrades observed on workload clusters when upgrading management cluster

Open
#3,522 2 comments 0 reactions 0 assignees View on GitHub
area/cli team/cli
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
Occasionally, changes such as the one in https://github.com/aws/eks-anywhere/pull/3402 can cause minor changes in the CAPI resources which the EKS-A controller picks up. When a management cluster is upgraded and a new EKS-A controller is installed on a cluster, it begins applying these new resources when reconciling not just the management cluster, _but the workload clusters as well_. This causes _all_ of them to perform an unexpected rollingupgrade. I believe this rolling upgrade for workload clusters is an undesirable behavior for users and should be avoided or mitigated.

When I was running some large scale tests with many workload clusters, I observed that every single workload cluster began its rolling upgrade simultaneously. If I have ten workload clusters with VM's of 16GB and 8 CPU's, and all of them start a rolling upgrade simultaneously, that will require a huge spike in resources which I may not be prepared to handle. This issue becomes even more prevalent with larger VM's and more workload clusters.

**What you expected to happen**:
Ideally, management cluster upgrades would never trigger rolling upgrades on workload clusters as a side effect. However, given that they share some resources, these upgrades should at least be staggered, and the CLI should wait for them all to complete and for the workload clusters to stabilize. Workload clusters should be paused for the duration of the management cluster's upgrade and unpaused gradually, while waiting for the cluster to become ready.

**How to reproduce it (as minimally and precisely as possible)**:
Upgrade a mgmt cluster with a workload cluster from v0.11.3 to v0.12.x (or the code in main at the time of writing of this issue). The management cluster upgrade will trigger a rolling upgrade for the workload cluster.

**Subtasks**:
- [x] https://github.com/aws/eks-anywhere/issues/3543
- [ ] https://github.com/aws/eks-anywhere/issues/3544

**Environment**:
- EKS Anywhere Release:main
- EKS Distro Release:

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the management-cluster upgrade from v0.11.3 to v0.12.x and trace how the EKS-A controller reconciles CAPI resources for workload clusters. Review the completed subtask in issue 3543 and the remaining work in issue 3544. Done means workload upgrades are avoided or safely staggered, with the CLI waiting for workload clusters to stabilize.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.