eksctl anywhere upgrade cluster doesn't use the extra hardware from hardware.csv
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
**What happened**:
I created an EKS anywhere cluster with 1 CP node, kubernetes 1.26
Then I try to upgrade it to 1.27, with 2 new CP nodes plus the existing CP node in the new hardware file.
cluster config file change is:
```
diff eksa-mgmt02-md-cluster.yaml eksa-mgmt02-md-cluster-sc.yaml
25c25
< kubernetesVersion: "1.26"
---
> kubernetesVersion: "1.27"
36c36
< osImageURL: "https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.26.7.gz"
---
> osImageURL: "https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.27.11.gz"
70c70
< IMG_URL: https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.26.7.gz
---
> IMG_URL: https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.27.11.gz
```
hardware change is:
```
diff hardware-mgmt02.csv hardware-mgmt02-cp3.csv
2a3,4
> eksa-control-02,...,type=cp,/dev/sda
> eksa-control-03,...,type=cp,/dev/sda
```
The command I use:
`eksctl anywhere upgrade cluster -f eksa-mgmt02-md-cluster-sc.yaml --hardware-csv hardware-mgmt02-cp3.csv --no-timeouts -v 9`
I see upgrade stuck with following errors:
```
...
2024-04-15T07:16:38.920Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1713165386699246825 kubectl get --ignore-not-found -o json --kubeconfig /home/armada/eksa/mgmt02/mgmt02/mgmt02-eks-a-cluster.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default mgmt02"}
2024-04-15T07:16:39.030Z V9 Cluster generation and observedGeneration {"Generation": 2, "ObservedGeneration": 2}
2024-04-15T07:16:39.031Z V5 Error happened during retry {"error": "cluster has an error: hardware validation failure: for rolling upgrade, minimum hardware count not met for selector '{\"type\":\"cp\"}': have 0, require 1", "retries": 1}
2024-04-15T07:16:39.031Z V5 Sleeping before next retry {"time": "1s"}
...
2024-04-15T07:32:06.179Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1713165386699246825 kubectl get --ignore-not-found -o json --kubeconfig /home/armada/eksa/mgmt02/mgmt02/mgmt02-eks-a-cluster.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default mgmt02"}
2024-04-15T07:32:06.285Z V9 Cluster generation and observedGeneration {"Generation": 2, "ObservedGeneration": 2}
2024-04-15T07:32:06.285Z V5 Error happened during retry {"error": "cluster has an error: hardware validation failure: for rolling upgrade, minimum hardware count not met for selector '{\"type\":\"cp\"}': have 0, require 1", "retries": 832}
2024-04-15T07:32:06.286Z V5 Sleeping before next retry {"time": "1s"}
...
```
**What you expected to happen**:
**How to reproduce it (as minimally and precisely as possible)**:
**Anything else we need to know?**:
**Environment**:
- EKS Anywhere Release: v0.18.7
- EKS Distro Release: 1.26/1.27
Contributor guide
Research direction
Start with the eksctl anywhere upgrade cluster entry point and compare eksa-mgmt02-md-cluster-sc.yaml with hardware-mgmt02-cp3.csv, using the reported -v 9 output and hardware validation failure. Trace how --hardware-csv is consumed during the upgrade; done means the added control-plane hardware is recognized and the upgrade no longer reports zero matching nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100