aws / aws/containers-roadmap

[EKS] [request]: Feature request to reduce the time it takes to terminate worker nodes in NotReady status

Open
#1,990 0 comments 7 reactions 0 assignees View on GitHub
EKS EKS Managed Nodes Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
This relates to Node in NotReady because Kubelet stopped posting node status to API Server.
When you delete such node or the Cluster Autoscaler delete such node, before the instance is terminated, Amazon EKS sends a signal to drain the pods from that node and then waits a few minutes or Amazon EKS lets Auto Scaling continue the termination of the instance after a few minutes. I observe that this takes between 15 minutes because of the 15 retry attempts every 60s.
e.g
```
Terminating EC2 instance: i-abc1234defg567 2023 March 27, 05:15:31 PM +01:00 2023 March 27, 05:32:50 PM +01:00
```
I am suggesting a feature request to reduce the retry attempts to not more than 2 minutes because this affects and delays new nodes to be added and pods that are supposed to be scheduled on another available nodes in the cluster. For critical pods in the cluster, this means about about 15 minutes downtime.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
To reduce possible workload downtime in EKS cluster caused by Nodes that becomes NotReady, marked for termination by ASG or Cluster Autoscaler but taking longer duration to terminate.

**What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem**.

I am trying to achieve any node marked as NoSchedule as a result of `node.kubernetes.io/unreachable` to be terminated as quickly as possible.

**Are you currently working around this issue?**
Going to EC2 to manually terminate the node from Console and Node is replaced immediately.
```
% date;aws ec2 terminate-instances --instance-id i-bbbb123456eee
Tue Mar 28 12:01:13 IST 2023
# check ASG history
Terminating EC2 instance: i-bbbb123456eee At 2023-03-28T11:02:20Z an instance was taken out of service in response to an EC2 health check indicating it has been terminated or stopped.
2023 March 28, 12:02:20 PM +01:00
2023 March 28, 12:03:05 PM +01:00
```

Contributor guide

Open the contributing guide

Research direction

Start with the EKS worker-node termination flow described in the issue, including the ASG, Cluster Autoscaler, EC2 termination, and pod-draining paths. Confirm how the 15 retry attempts at 60-second intervals are applied; done means reducing the retry window to no more than two minutes while preserving node replacement and workload draining behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.