[EKS] [request]: ASG is not respecting to PDB during Nodegroup nodes version update when NodegroupScalingConfig is modified by aws-cluster-autoscaler
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Based on our investigation and the official AWS documentation [here](https://docs.aws.amazon.com/eks/latest/userguide/update-managed-node-group.html)
, we found that the aws-cluster-autoscaler—an essential baseline service in our EKS Service Catalog—may adjust the ASG's desired capacity during NodeGroup updates. This behaviour can result in the ASG bypassing the PDB constraints, as also acknowledged by AWS.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
During our ongoing discussions with AWS Support, we identified that the issue stems from a conflict between the AWS Cluster Autoscaler and the AWS Auto Scaling Group (ASG). Specifically, any modifications to the NodegroupScalingConfig cause the ASG to bypass Pod Disruption Budget (PDB) constraints — a behavior acknowledged by AWS.
The current workaround recommended by AWS during node version updates is to temporarily scale the Cluster Autoscaler down to zero. However, the Cluster Autoscaler is a critical component of our Service Catalog Product, and disabling it — even temporarily — is not an acceptable option for our production environment.
We are therefore requesting a long-term solution that resolves the conflict between the Cluster Autoscaler and ASG, ensuring that PDBs are always respected during node updates and scaling operations.
**Are you currently working around this issue?**
To mitigate the impact, we have temporarily increased the number of application replicas to reduce the risk of downtime during node updates, in case PDBs are not respected.
**Additional context**
After following up with AWS Support, we received the following suggestions:
1- One of the methods to avoid such conflict (between ASG and aws-cluster-autoscaler) would be to disable the autoscaler before the starting the upgradation process of the managed node group.
2- Another, way to avoid such issues in the future is to make sure that the pod disruption budget that you are using for your application is configured correctly. Also, you could try increasing the replicas count of the deployment.
Regarding suggestion 1:
We believe scaling down the aws-cluster-autoscaler to 0 during updates is not a suitable solution for our EKS Service Catalog Product.
This service is considered baseline, and we prefer to avoid disabling it during the Bottlerocket upgrade.
Regarding suggestion 2:
The application team has already updated the deployment configuration to 3 replicas with minAvailable: 1 in the PDB.
However, we are not fully convinced that this change (suggestion 2) alone will prevent the issue, because the core problem seems to be the conflict between the cluster autoscaler and the ASG during updates.
**Attachments**
Contributor guide
Research direction
Start with the linked AWS documentation on updating managed node groups and review the reported interaction between Cluster Autoscaler, ASG capacity changes, and Pod Disruption Budgets. The issue does not identify repository files, tests, or an implementation entry point, and completion criteria for a project-side change are not defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100