[EKS] [request]: Managed Node Group Deletion Fails if Node Drain Fails
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
When upgrading an EKS Managed Node Group, there is an option for a "rolling update" or a "force update". During a "rolling update" if a node fails to drain completely, the update fails. Draining might fail because of PodDisruptionBudget configurations, post hooks, or other cluster/pod/deployment settings. Sometimes this is desirable, sometimes it is not desirable and so the "force update" option allows the upgrade to continue despite these drain failures.
However, when deleting a managed node group, there is no similar option. Once a deletion is started, there is no stopping it (essentially the "force" behavior is the only behavior). Much like upgrades though, there are situations where we want the deletion to fail if a node fails to drain completely. We request that this option be added for managed node group deletion, and for it to be the default behavior similar to upgrades.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When we make changes to EKS Managed Node Groups such as updating the instance types we are using, the process creates a new managed node group and then deletes the old managed node group. We've had multiple occasions where this causes an outage.
In some cases, draining the old nodes took much longer than the [undefined "few minutes"](https://docs.aws.amazon.com/eks/latest/userguide/delete-managed-node-group.html) which EKS waits for the drain to finish, generally because of PodDisruptionBudgets which we have in place to ensure availability of our applications. In another case, the new managed node group was unable to bring up sufficient instances to allow all pods to reschedule, essentially making the cluster unusable until we could undo the change.
**Are you currently working around this issue?**
We are avoiding updates that would require us to create and then delete a managed node group, which is the main situation in which we would want the deletion to fail. If we need to do this though, we will have to 1) create the new node group, 2) cordon off the old node group, 3) manually drain all the nodes, 4) delete the old node group assuming the drain was successful. This manual process is very undesirable for us, especially in our larger clusters. We might consider going back to managing autoscaling groups ourselves, and use lambda functions/lifecycle hooks to ensure this behavior as we did before adopting managed node groups.
**Additional context**
Because the managed node group deletion behavior is totally different from the managed node group upgrade behavior, it took us quite a while to track down why we were seeing an outage during these events. I believe this is a bug, since it differs significantly from the behavior of upgrades and does not respect the resiliency and availability settings that Kubernetes allows us to use to prevent outages of this type (i.e. PodDisruptionBudgets).
**Attachments**
Contributor guide
Research direction
Start by comparing the EKS managed node group deletion behavior described here with the existing rolling and force update options. The issue does not name repository files, tests, or implementation entry points; done would require an agreed deletion option that fails when draining fails, with the requested default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100