[EKS] [NodeGroups] [request]: Node Groups take 30 minutes to rotate nodes without reason
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
When we do any updates to Managed Node Group Templates, it's taking nearly 30 minutes to apply any change.
For example, if you update tags on Node Group which involves changes to template and replacing instances (not sure why we need to replace instances if we add tags?) it will take around 25 to 30 minutes to replace those instances.
This is a huge amount of time for such basic change.
Another problem is that in my test cluster where I have just 2 nodes (development and testing) such change will create additional 4 nodes. Those nodes will be in place for around 30 minutes since that change takes so long time.
This means during a simple update to tag names or values, we are paying for 4 additional instances without any reason.
**Which service(s) is this request for?**
EKS Node Groups
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
The problem seems to be with bad or delayed communication between AWS and EKS Control Plane. I was looking at this issue some time ago with AWS Support and noticed that:
- We issue tag change or any other change which require update to template version
- EKS goes and tries to update Node Groups and replace nodes inside that Node Group
- During above process which takes around 30 minutes with 1 or 2 nodes I see no communication from AWS to EKS. Nodes are not being cordoned, nor any pods evicted or asking to be evicted. This would mean that there is an issue with Node Draining
- Usually after 20+ minutes all of a sudden we start seeing node draining process start
- Once drain is completed, instances are remove one by one from Auto Scaling Group
I was trying to use [Node Termination Handler](https://github.com/aws/aws-node-termination-handler) to help me with this issue. IMDS setup doesn't seem to do anything and other option will require creation of Event Bridge and other resources, which seems like an overkill for something that Node Group was meant to provide by default.
I haven't yet tested Queue option so I'm not even sure if this works. I can only confirm that default IMDS option does nothing.
**Are you currently working around this issue?**
There is no way to solve this problem
**Additional context**
Anything else we should know?
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
Start by reviewing the reported EKS Managed Node Groups update and node-draining sequence, including the interaction with Node Termination Handler described in the issue. Done would require identifying and addressing the delayed draining and unnecessary temporary capacity, but no repository files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100