[EKS] [managed node group drain pods due to AZRebalancing]: AZRebalancing is automatically applied, so cannot stop pods from draining in MNG.
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Feature request allowing to switch to only cordon on AZ Rebalance and EC2 capacity rebalance on Managed EKS node group.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
>>The main problem is actually the inability to prevent the draining of nodes on certain notifications (AZ Rebalance and EC2 Capacity rebalance).
UseCase: "Gitlab Runner cost reduction while maximizing throughput"
Gitlab spins up bare pods for each CICD job in its Kubernetes Executor (https://docs.gitlab.com/runner/executors/kubernetes.html). Since these are bare pods, these will no survive the draining of the node on which they are scheduled resulting in a failed job in Gitlab.
Since these jobs can be restarted if necessary we are using spot instances for cost reduction. We do want to optimize for throughput instead of maximum availability so nodes should only be drained when its absolutely necessary (eg Spot termination notification). Otherwise we want to leave these pods running as long as possible.
**Are you currently working around this issue?**
How are you currently solving this problem?
1. Update the underlying ASG out of band to disable AZ rebalancing. It is not recommended as per standards.
2. Creating ONE nodegroup per AZ and split the original nodegroup capacity among those nodegroups. This would result in THREE nodegroups that only operate in a single ASG, which would mean that the rebalancing would never happen.
**Additional context**
With EKS managed node groups we can't control this behavior, like we can in the node termination handler (https://github.com/aws/aws-node-termination-handler/tree/main/config/helm/aws-node-termination-handler) called `enableRebalanceDraining`, resulting in many unnecessary drained nodes and failed Gitlab jobs. It would be nice to have this option in EKS managed node groups.
**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
Contributor guide
Research direction
Start by reviewing the issue's EKS Managed Node Groups request and the linked node termination handler reference, then compare the two documented workarounds. Done would require an agreed EKS capability for controlling AZ Rebalance and EC2 Capacity Rebalance draining; the payload names no repository files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100