aws / aws/eks-anywhere

eksctl anywhere operations rendered unusable after osImageURL change

Open
#9,899 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

This is anything but nice. It completely breaks `eksctl anywhere` operations just because of a valid URL change.

---

### Context

- Setup is **Bare Metal** in a test environment that probably gets wiped 10 times per day, if not more.
- You created a new EKS-A cluster with **3 control plane nodes** and **1 worker node**.

hardware.csv

```csv
hostname,mac,ip_address,netmask,gateway,nameservers,labels,disk
cplane-0,XX:XX:XX:XX:XX:01,10.162.10.131,255.255.255.240,10.162.10.129,8.8.8.8|8.8.4.4,type=cp,/dev/sda
cplane-1,XX:XX:XX:XX:XX:02,10.162.10.132,255.255.255.240,10.162.10.129,8.8.8.8|8.8.4.4,type=cp,/dev/sda
cplane-2,XX:XX:XX:XX:XX:03,10.162.10.133,255.255.255.240,10.162.10.129,8.8.8.8|8.8.4.4,type=cp,/dev/sda
worker-0,XX:XX:XX:XX:XX:04,10.162.10.134,255.255.255.240,10.162.10.129,8.8.8.8|8.8.4.4,type=worker,/dev/sda
worker-1,XX:XX:XX:XX:XX:05,10.162.10.135,255.255.255.240,10.162.10.129,8.8.8.8|8.8.4.4,type=worker,/dev/sda
worker-2,XX:XX:XX:XX:XX:06,10.162.10.136,255.255.255.240,10.162.10.129,8.8.8.8|8.8.4.4,type=worker,/dev/sda
```

---

### Cluster creation command

```bash
eksctl anywhere create cluster \
--hardware-csv hardware.csv \
-f eksa-mgmt-cluster.yaml \
--no-timeouts
```

The cluster is up (with 1 worker node):

```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
cplane-0 Ready control-plane 19m v1.31.9-eks-ca3410b
cplane-1 Ready control-plane 15m v1.31.9-eks-ca3410b
cplane-2 Ready control-plane 9m53s v1.31.9-eks-ca3410b
worker-0 Ready 3m46s v1.31.9-eks-ca3410b
```

---

### You now want to add more nodes

It doesn't really matter if you created the cluster with 1, 2 or 20 nodes — the idea is that you want to scale up the workers. You still have 2 "spares" in `hardware.csv` so you only have to increase the count.

> Oh, and by the way, you re-organized the S3 bucket that serves the images and now the image is in a path pointing to eks anywhere bundle version

```diff
--- orig.yaml 2025-07-13 10:29:49.840935788 +0000
+++ new.yaml 2025-07-13 10:30:22.217130563 +0000
@@ -28,7 +28,7 @@
managementCluster:
name: eks-a
workerNodeGroupConfigurations:
- - count: 1
+ - count: 3
machineGroupRef:
kind: TinkerbellMachineConfig
name: eks-a
@@ -41,7 +41,7 @@
metadata:
name: eks-a
spec:
- osImageURL: "https://BUCKET_NAME.s3.eu-central-1.amazonaws.com/ubuntu-2204-kube-1-31.gz"
+ osImageURL: "https://BUCKET_NAME.s3.eu-central-1.amazonaws.com/v0.22.6/ubuntu-2204-kube-1-31.gz"
tinkerbellIP: "10.162.10.141"

---
```

---

### Start the cluster upgrade to add more nodes

```bash
eksctl anywhere upgrade cluster \
--hardware-csv hardware.csv \
-f cluster.yaml \
--no-timeouts
```

---

### Preflight check passes nicely

```
Performing setup and validations
✅ Tinkerbell provider validation
✅ SSH Keys present
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Control plane ready
✅ Worker nodes ready
✅ Nodes ready
✅ Cluster CRDs ready
✅ Cluster object present on workload cluster
✅ Upgrade cluster kubernetes version increment
✅ Upgrade cluster worker node group kubernetes version increment
✅ Validate authentication for git provider
✅ Validate immutable fields
✅ Validate cluster's eksaVersion matches EKS-Anywhere Version
✅ Validate eksa controller is not paused
✅ Validate extended kubernetes version support is supported
✅ Validate pod disruption budgets
✅ Validate eksaVersion skew is one minor version
Ensuring etcd CAPI providers exist on management cluster before upgrade
Pausing GitOps cluster resources reconcile
Upgrading core components
```

---

Well, that's about all. The process quickly starts throwing this error, endlessly, in a loop, with `--no-timeouts`. The message **makes absolutely no sense**. It looks like it wants to start a rolling upgrade when you change the image path and not the version

```
"Cluster generation and observedGeneration","Generation":2,"ObservedGeneration":2
"Error happened during retry","error":"cluster has an error: hardware validation failure: for node rollout, minimum hardware count not met for selector '{\"type\":\"cp\"}': have 0, require 1","retries":138
"Sleeping before next retry","time":"1s"
```

[osImageURL](https://anywhere.eks.amazonaws.com/docs/getting-started/baremetal/bare-spec/#osimageurl-required) documentation states:

```
The osImageURL must contain the Cluster.Spec.KubernetesVersion or Cluster.Spec.WorkerNodeGroupConfiguration[].KubernetesVersion version (in case of modular upgrade).
For example, if the Kubernetes version is 1.31, the osImageURL name should include 1.31, 1_31, 1-31 or 131.
```

So, why is it trying to upgrade just based on a simple path change in the image URL? The **required** part (`1-31`) is still there and it does not differ from `kubernetesVersion`

---

### The fix?

Obvious: put back the old image URL and start over... oh, wait! You need to stop the existing process. Ok, CTRL+C it, change the image to the old path and start again.

---

### Check the emojis again:

```
Performing setup and validations
✅ Tinkerbell provider validation
✅ SSH Keys present
✅ Validate OS is compatible with registry mirror configuration
✅ Validate certificate for registry mirror
✅ Control plane ready
❌ Validation failed {"validation": "worker nodes ready", "error": "machine deployment is in ScalingUp phase", "remediation": "ensure machine deployments for cluster eks-a are ready"}
✅ Nodes ready
✅ Cluster CRDs ready
✅ Cluster object present on workload cluster
✅ Upgrade cluster kubernetes version increment
✅ Upgrade cluster worker node group kubernetes version increment
✅ Validate authentication for git provider
✅ Validate immutable fields
✅ Validate cluster's eksaVersion matches EKS-Anywhere Version
✅ Validate eksa controller is not paused
✅ Validate extended kubernetes version support is supported
✅ Validate pod disruption budgets
✅ Validate eksaVersion skew is one minor version
Error: failed to upgrade cluster: validations failed: machine deployment is in ScalingUp phase
```

---

Great. Now I can't add nodes or perform any `eksctl anywhere` operation whatsoever.

> As you can see, it is not enough to just go back to the old image to fix this, it implies some careful fiddling with the cluster CRs.
> I am not going to detail this here because people might think it could apply in other situations and they could potentially break a production cluster.

---

Please address this.
LE: I think it should actually throw a clearer error since scale up and upgrades are not supported in the same time

---

**Environment**:
- EKS Anywhere Release
```
$ eksctl anywhere version
Version: v0.22.6
Release Manifest URL: https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml
Bundle Manifest URL: https://anywhere-assets.eks.amazonaws.com/releases/bundles/100/manifest.yaml
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the bare-metal scenario with hardware.csv, cluster.yaml, the two osImageURL values, and the eksctl anywhere upgrade cluster command. Start by examining the upgrade and worker scaling entry points and the reported hardware-validation and ScalingUp phases. Done means an image-path-only change does not incorrectly trigger an upgrade, or the command reports clearly when scaling and upgrades cannot run together.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.