aws / aws/eks-anywhere

fail to scale EKS anywhere cluster for bare metal

Open
#7,614 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
I try to scale my cluster (from 1 cp node to 3 cp nodes out of 4 cp nodes, hardware.csv has 4 nodes, and I want to use 3 of them as cp nodes) with command:
```
eksctl anywhere upgrade cluster
-f cluster.yaml
--hardware-csv hardware.csv
--kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig
```
If I keep cp count 1 in the cluster.yaml, the command completes in 2 minutes, however, nothing change, even I don't see new hardware with follow command:
`kubectl get hardware -n eksa-system --show-labels `

If I change cp count to 3 in the cluster.yaml, I keep seeing this message:
```
2024-02-15T17:26:49.783Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1708017136282361003 kubectl get --ignore-not-found -o json --kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default mgmt02"}
2024-02-15T17:26:49.884Z V9 Cluster generation and observedGeneration {"Generation": 2, "ObservedGeneration": 1}
2024-02-15T17:26:49.884Z V5 Error happened during retry {"error": "cluster generation (2) and observedGeneration (1) differ", "retries": 782}
2024-02-15T17:26:49.884Z V5 Sleeping before next retry {"time": "1s"}
```

**What you expected to happen**:
I expect the command to succeed with new hardware added

**How to reproduce it (as minimally and precisely as possible)**:

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: v0.18.2
- EKS Distro Release: 1.26

Contributor guide

Open the contributing guide

Research direction

Start with the upgrade cluster command and the provided cluster.yaml and hardware.csv, then inspect the kubectl hardware output and retry logs. Determine why scaling from one to three control-plane nodes leaves hardware unchanged or keeps Cluster generation ahead of observedGeneration; done means the upgrade succeeds and new hardware is added.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.