[EKS] [request]: Improve Health checking if Managed Nodegroup failed to join EKS cluster at creation stage
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Currently, Managed Nodegroup doesn't check the Health status again if instances failed to join EKS cluster at creation stage. Hope it can improve the Health Issue checking(Like `5 minutes` checking again), or, improve the error message to mention about user have to delete nodegroup failed at creation then create it again.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When the first time I created the managed nodegroup, for some reason I can aware my managed nodegroup unable to join the EKS cluster and it shows error `Instances failed to join the kubernetes cluster` in the Health Issue.
After fixing the problem, I notice that the status won't change again even the instances correctly join the k8s cluster and able to schedule Pods. It makes me confused.
**Are you currently working around this issue?**
The status unable to be changed. You have to delete the managed nodegroup and recreate a new one.
**How to reproduce**
You can follow steps to replicate the issue:
```bash
[Configurations]
------------------------------
# Cluster configurations
CLUSTER=test
CLUSTER_SG=$(aws eks describe-cluster --name $CLUSTER --region us-west-2 | jq -r ".cluster.resourcesVpcConfig.clusterSecurityGroupId")
# Nodegroup configurations
SUBNETS="subnet-AAAAAAAAAA subnet-BBBBBBBB subnet-CCCCCCCC"
NODE_ROLE=arn:aws:iam::XXXXXXXX:role/EKSManagedNodeWorkerRole
NODE_GROUP_NAME=box
------------------------------
```
1) Temporary remove the connectivity of security group used by control plane and nodegroup
```bash
$ aws ec2 revoke-security-group-egress --group-id $CLUSTER_SG --protocol all --port all --cidr 0.0.0.0/0 --region us-west-2
```
2) Create managed nodegroup
```bash
$ aws eks create-nodegroup --cluster-name $CLUSTER --nodegroup-name $NODE_GROUP_NAME --subnets $SUBNETS --node-role $NODE_ROLE --region us-west-2
```
3) Monitor the nodegroup, wait about ~10minutes, will see the error message
```bash
$ while true; do aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name $NODE_GROUP_NAME --region us-west-2 | jq -r ".nodegroup.health.issues"; sleep 3; done
[
{
"resourceIds": [
"i-AAAAAAAAAAAAAA",
"i-BBBBBBBBBBBBBB"
],
"message": "Instances failed to join the kubernetes cluster",
"code": "NodeCreationFailure"
}
]
```
4) Then, recover the netework setting
```bash
$ aws ec2 authorize-security-group-egress --group-id $CLUSTER_SG --protocol all --port all --cidr 0.0.0.0/0 --region us-west-2
```
5) Can see the Nodes and become `Ready`
```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-57-130.us-west-2.compute.internal Ready 2m14s v1.14.8-eks-b8860f
ip-192-168-68-87.us-west-2.compute.internal Ready 2m11s v1.14.8-eks-b8860f
```
6) The Health Issue still show the error and status did not flip out like ACTIVE or other status (Keep to be CREATE_FAILED)
```bash
$ aws eks describe-nodegroup --cluster-name $CLUSTER --nodegroup-name $NODE_GROUP_NAME --region us-west-2
{
"nodegroup": {
"status": "CREATE_FAILED",
...
"nodegroupName": "box",
"nodegroupArn": "arn:aws:eks:us-west-2:XXXXXXXX:nodegroup/test/box/fcb83117-9467-b7bf-e6e6-XXXXXXXX",
"health": {
"issues": [
{
"resourceIds": [
"i-AAAAAAAAAAAAAA",
"i-BBBBBBBBBBBBBB"
],
"message": "Instances failed to join the kubernetes cluster",
"code": "NodeCreationFailure"
}
]
},
...
}
}
```
Contributor guide
Research direction
Reproduce the EKS managed nodegroup flow using the supplied aws ec2, aws eks, jq, and kubectl commands; first compare the CREATE_FAILED health issue with nodes becoming Ready after connectivity is restored. Done means the health/status behavior is updated or the error clearly explains that the nodegroup must be deleted and recreated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100