[EKS] [request]: Configurable or shorter delay for Node Auto Repair node termination
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
As confirmed by AWS support, EKS Node Auto Repair waits 30 to 40 minutes before removing a failed node. This is way to long, I would love to see this either very short (few minutes) or configurable.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
At current, if a node fails to report (Kubelet not responding), the pods on it are stuck in Terminating until the node disappears, as the control plane has no way to know the actual state of the process. This holds statefulset pods hostage, especially with EBS persistent volumes attached, which can only be attached to a new node if the old attachment is detached.
In a well-designed archiecture with redundancy, it is better to terminate a misbehaving host fast and fail over, than to wait overly long for a possible (but often unlikely) recovery.
**Are you currently working around this issue?**
We terminate affected hosts (we have lots of them ATM for another reason) manually, and think about implementing our own "termination routine" (get all NotReady nodes and terminate them every 4 minutes or so).
**Additional context**
Already consulted AWS support, they recommended bringing the issue here. Case ID: 174471733700573
Contributor guide
Research direction
This is an EKS service feature request rather than a repository code change, and the issue mentions no files, tests, or implementation entry points. Start by reviewing the requested shorter or configurable Node Auto Repair delay and the described effects on NotReady nodes, terminating pods, and EBS volume detachment; done means AWS provides and documents the requested behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100