aws / aws/eks-anywhere

1st CP node always fails `MachineHealthCheck` when creating a Bottlerocket workload cluster

Open
#3,345 0 comments 0 reactions 0 assignees View on GitHub
area/providers/capv kind/bug
Dominant language
Go
Stars
2.1k
Forks
328
Avg merge
1d 4h
Merged PRs (30d)
9

Description

**What happened**:
When creating a 3 control plane Bottlerocket EKS Anywhere cluster from an existing management, the first control plane node always fails because of MachineHealthCheck failure and gets deleted and re-provisioned.
Once the node gets re-provisioned, the cluster is in a proper functioning state but it is still strange that the first control plane node always fails. Another strange thing is that this happens right about when EKS-A reports the the cluster is successfully created.
We tried this several times and saw the same behavior over and over.

**How to reproduce it (as minimally and precisely as possible)**:
Create a Bottlerocket management cluster. Using this management cluster, attempt to create a 3 control plane Bottlerocket workload cluster and wait till the CLI finishes. Once it's done, check the machine status on the management cluster. It should show one node as deleting.
I also had AWS IAM Authenticator deployed on both the management and workload cluster for this test. Not sure whether this matters or not but wanted to mention it.

**Environment**:
- EKS Anywhere Release: v0.11.1
- EKS Distro Release: v1.23

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with a Bottlerocket management cluster and a three-control-plane workload cluster, then inspect MachineHealthCheck and machine status on the management cluster as the CLI completes. Done means the first control-plane node remains healthy after creation and no node is deleted and re-provisioned.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.