aws / aws/eks-anywhere

eksctl anywhere upgrade cluster doesn't use the extra hardware from hardware.csv

Open
#7,818 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
I created an EKS anywhere cluster with 1 CP node, kubernetes 1.26

Then I try to upgrade it to 1.27, with 2 new CP nodes plus the existing CP node in the new hardware file.

cluster config file change is:
```
diff eksa-mgmt02-md-cluster.yaml eksa-mgmt02-md-cluster-sc.yaml
25c25
< kubernetesVersion: "1.26"
---
> kubernetesVersion: "1.27"
36c36
< osImageURL: "https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.26.7.gz"
---
> osImageURL: "https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.27.11.gz"
70c70
< IMG_URL: https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.26.7.gz
---
> IMG_URL: https://.blob.core.windows.net/ubuntu-2004-efi/ubuntu-2004-efi-eksa-kube-v1.27.11.gz
```

hardware change is:
```
diff hardware-mgmt02.csv hardware-mgmt02-cp3.csv
2a3,4
> eksa-control-02,...,type=cp,/dev/sda
> eksa-control-03,...,type=cp,/dev/sda
```
The command I use:
`eksctl anywhere upgrade cluster -f eksa-mgmt02-md-cluster-sc.yaml --hardware-csv hardware-mgmt02-cp3.csv --no-timeouts -v 9`

I see upgrade stuck with following errors:
```
...
2024-04-15T07:16:38.920Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1713165386699246825 kubectl get --ignore-not-found -o json --kubeconfig /home/armada/eksa/mgmt02/mgmt02/mgmt02-eks-a-cluster.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default mgmt02"}
2024-04-15T07:16:39.030Z V9 Cluster generation and observedGeneration {"Generation": 2, "ObservedGeneration": 2}
2024-04-15T07:16:39.031Z V5 Error happened during retry {"error": "cluster has an error: hardware validation failure: for rolling upgrade, minimum hardware count not met for selector '{\"type\":\"cp\"}': have 0, require 1", "retries": 1}
2024-04-15T07:16:39.031Z V5 Sleeping before next retry {"time": "1s"}
...
2024-04-15T07:32:06.179Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1713165386699246825 kubectl get --ignore-not-found -o json --kubeconfig /home/armada/eksa/mgmt02/mgmt02/mgmt02-eks-a-cluster.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default mgmt02"}
2024-04-15T07:32:06.285Z V9 Cluster generation and observedGeneration {"Generation": 2, "ObservedGeneration": 2}
2024-04-15T07:32:06.285Z V5 Error happened during retry {"error": "cluster has an error: hardware validation failure: for rolling upgrade, minimum hardware count not met for selector '{\"type\":\"cp\"}': have 0, require 1", "retries": 832}
2024-04-15T07:32:06.286Z V5 Sleeping before next retry {"time": "1s"}
...
```

**What you expected to happen**:

**How to reproduce it (as minimally and precisely as possible)**:

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: v0.18.7
- EKS Distro Release: 1.26/1.27

Contributor guide

Open the contributing guide

Research direction

Start with the eksctl anywhere upgrade cluster entry point and compare eksa-mgmt02-md-cluster-sc.yaml with hardware-mgmt02-cp3.csv, using the reported -v 9 output and hardware validation failure. Trace how --hardware-csv is consumed during the upgrade; done means the added control-plane hardware is recognized and the upgrade no longer reports zero matching nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.