aws / aws/containers-roadmap

[EKS] [request]: Improve Health checking if Managed Nodegroup failed to join EKS cluster at creation stage

Open
#764 5 comments 66 reactions 0 assignees View on GitHub
EKS EKS Managed Nodes Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Currently, Managed Nodegroup doesn't check the Health status again if instances failed to join EKS cluster at creation stage. Hope it can improve the Health Issue checking(Like `5 minutes` checking again), or, improve the error message to mention about user have to delete nodegroup failed at creation then create it again.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When the first time I created the managed nodegroup, for some reason I can aware my managed nodegroup unable to join the EKS cluster and it shows error `Instances failed to join the kubernetes cluster` in the Health Issue.

After fixing the problem, I notice that the status won't change again even the instances correctly join the k8s cluster and able to schedule Pods. It makes me confused.

**Are you currently working around this issue?**
The status unable to be changed. You have to delete the managed nodegroup and recreate a new one.

**How to reproduce**

You can follow steps to replicate the issue:

```bash
[Configurations]
------------------------------
# Cluster configurations
CLUSTER=test
CLUSTER_SG=$(aws eks describe-cluster --name $CLUSTER --region us-west-2 | jq -r ".cluster.resourcesVpcConfig.clusterSecurityGroupId")

# Nodegroup configurations
SUBNETS="subnet-AAAAAAAAAA subnet-BBBBBBBB subnet-CCCCCCCC"
NODE_ROLE=arn:aws:iam::XXXXXXXX:role/EKSManagedNodeWorkerRole
NODE_GROUP_NAME=box
------------------------------
```

1) Temporary remove the connectivity of security group used by control plane and nodegroup

```bash
$ aws ec2 revoke-security-group-egress --group-id $CLUSTER_SG --protocol all --port all --cidr 0.0.0.0/0 --region us-west-2
```

2) Create managed nodegroup
```bash
$ aws eks create-nodegroup --cluster-name $CLUSTER --nodegroup-name $NODE_GROUP_NAME --subnets $SUBNETS --node-role $NODE_ROLE --region us-west-2
```

3) Monitor the nodegroup, wait about ~10minutes, will see the error message

```bash
$ while true; do aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name $NODE_GROUP_NAME --region us-west-2 | jq -r ".nodegroup.health.issues"; sleep 3; done

[
{
"resourceIds": [
"i-AAAAAAAAAAAAAA",
"i-BBBBBBBBBBBBBB"
],
"message": "Instances failed to join the kubernetes cluster",
"code": "NodeCreationFailure"
}
]
```

4) Then, recover the netework setting

```bash
$ aws ec2 authorize-security-group-egress --group-id $CLUSTER_SG --protocol all --port all --cidr 0.0.0.0/0 --region us-west-2
```

5) Can see the Nodes and become `Ready`

```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-57-130.us-west-2.compute.internal Ready 2m14s v1.14.8-eks-b8860f
ip-192-168-68-87.us-west-2.compute.internal Ready 2m11s v1.14.8-eks-b8860f
```

6) The Health Issue still show the error and status did not flip out like ACTIVE or other status (Keep to be CREATE_FAILED)

```bash
$ aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name $NODE_GROUP_NAME --region us-west-2
{
"nodegroup": {
"status": "CREATE_FAILED",
...
"nodegroupName": "box",
"nodegroupArn": "arn:aws:eks:us-west-2:XXXXXXXX:nodegroup/test/box/fcb83117-9467-b7bf-e6e6-XXXXXXXX",
"health": {
"issues": [
{
"resourceIds": [
"i-AAAAAAAAAAAAAA",
"i-BBBBBBBBBBBBBB"
],
"message": "Instances failed to join the kubernetes cluster",
"code": "NodeCreationFailure"
}
]
},
...
}
}
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the EKS managed nodegroup flow using the supplied aws ec2, aws eks, jq, and kubectl commands; first compare the CREATE_FAILED health issue with nodes becoming Ready after connectivity is restored. Done means the health/status behavior is updated or the error clearly explains that the nodegroup must be deleted and recreated.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.