aws / aws/eks-anywhere

upgrade kubernetes version of EKSA cluster for bare metal with 2 CP nodes (1 used + 1 idle) doesn't work

Open
#7,820 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
It's ok for me to upgrade a cluster with 4 CP nodes (3 used + 1 idle).

However, when I try to upgrade a cluster with 2 CP nodes (1 used + 1 idle), the upgrade stuck after the idle node is "Provisioned":

```
armada@admin-machine2:~/eksa/mgmt02$ kubectl get workflow -A -o wide
NAMESPACE NAME TEMPLATE STATE
eksa-system mgmt02-standalone2-control-plane-template-1710122858425-44xpk mgmt02-standalone2-control-plane-template-1710122858425-44xpk STATE_SUCCESS

armada@admin-machine2:~/eksa/mgmt02$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
eksa-control-02 Ready control-plane 5h16m v1.26.14 10.20.22.224 Ubuntu 20.04.6 LTS 5.4.0-172-generic containerd://1.7.10
eksa-wk-650-01 Ready 4h56m v1.26.14 10.20.22.227 Ubuntu 20.04.6 LTS 5.4.0-172-generic containerd://1.7.10

armada@admin-machine2:~/eksa/mgmt02$ kubectl get machines.cluster.x-k8s.io -A -o wide
NAMESPACE NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION
eksa-system mgmt02-standalone2-b94wm mgmt02-standalone2 eksa-control-02 tinkerbell://eksa-system/eksa-control-02 Running 4h54m v1.26.10-eks-1-26-21
eksa-system mgmt02-standalone2-md-0-dvxsx-ntvph mgmt02-standalone2 eksa-wk-650-01 tinkerbell://eksa-system/eksa-wk-650-01 Running 4h54m v1.26.10-eks-1-26-21
eksa-system mgmt02-standalone2-p8fwl mgmt02-standalone2 tinkerbell://eksa-system/eksa-main-control-01 Provisioned 45m v1.27.7-eks-1-27-15
```

An smoking issue after the idle node is "Provisioned" is that, when I run above "kubectl get ..." commands, I may see such error message:

```
...
E0311 07:45:31.895702 953673 memcache.go:265] couldn't get current server API group list: Get "https://10.20.22.222:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

```
**What you expected to happen**:

**How to reproduce it (as minimally and precisely as possible)**:

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: v0.18.2
- EKS Distro Release: 1.26/1.27

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the upgrade on the bare-metal EKS Anywhere v0.18.2 setup with two control-plane nodes, using the reported kubectl get workflow, node, and machines.cluster.x-k8s.io commands. Inspect the control-plane machine after it reaches Provisioned and capture the cluster state and certificate errors. Done means the upgrade completes with one active and one idle control-plane node, and kubectl access remains valid.

Written by the indexing model from the issue text.

Assessment

Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.