aws / aws/containers-roadmap

[EKS] [bug]: auto-scaling group ends up in a bad state after `kubectl delete node`

Open
#1,811 6 comments 40 reactions 0 assignees View on GitHub
EKS EKS Managed Nodes Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Currently whenever `kubectl delete node` command is ran in the cluster - node is removed from k8s, but the EC2 instance behind the node is not terminated. As a result AWS auto-scaling group behind k8s node group does not create new EC2 instances, which also breaks things like cluster auto-scaler.

An example would look like this:
* In the initial setup you have ASG with 5 EC2 instances (desired size is 5), all onboarded as nodes on k8s cluster.
* `kubectl detele node` command runs in the cluster, removing a single node
* ASG still has "desired size = 5", yet opening the "nodes" tab you can see only 4 nodes.
* Since a node was removed, auto-scaling controller may decided to ask for an additional node to be created (e.g. to handle scale-up)
* Yet this request *would not be handed by ASG*, because according to it there are already 5 nodes available.

The only way I know of how to resolve the situation is to MANUALLY find out EC2 instance that is no longer mapped to a node in k8s cluster and terminate it, then ASG would pick this information up and continue handling auto-scaler requests.

**Which service(s) is this request for?**
EKS, ASG

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
There is no particular need to use `kubectl delete node`, but having this behavior in the system is very dangerous. I ended up in this situation because I wanted to get rid of nodes that seemed to be poisoned (pods running on these were performing worse than pods of same service running on all other nodes in the cluster) - it turned out the issue was totally unrelated, but in doing `kubectl delete node` I messed up the cluster and put it into a bad state that required a fair amount of effort to get to the bottom of.

**Are you currently working around this issue?**
Yes, manually deleting EC2 instance is a viable workaround

**Additional context**
You can see more details in:
* this StackOverflow thread where a different person stumbled on this problem before: https://stackoverflow.com/questions/57554812/my-nodes-got-deleted-in-eks-how-can-i-recover
* In this support request where there are steps from support engineer who reproduced the problem: https://support.console.aws.amazon.com/support/home?region=us-east-1#/case/?displayId=10511080081

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction sequence in the issue: run `kubectl delete node` in an EKS cluster with an ASG-backed node group and compare the node and EC2 instance states. Read the linked Stack Overflow thread and AWS Support case for the reported behavior and reproduction details. Done would require an agreed service-level fix or documented behavior that prevents the ASG and cluster from becoming inconsistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.