EKS: autoscaler: allow scaleout when adding pods would exceed number of available IPs on a node
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I ran into similar errors today as these issues:
- https://github.com/kubernetes/autoscaler/issues/2210
- https://github.com/aws/amazon-vpc-cni-k8s/issues/1245
TLDR:
> (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "redacted" network for pod "redacted": NetworkPlugin cni failed to set up pod "redacted" network: add cmd: failed to assign an IP address to container
The suggestion there was to make an issue here for a feature request. I did not find one by searching all issues for "autoscaler ip", hence this request.
It seems like the error in these threads and in my EKS cluster is due to a pod being deployed onto a node that has no more network interfaces available on it to support the pods.
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I made 3 deployments, each with 1 pod, via our CI/CD pipeline, and one out of 3 came back with this error. It is the first time I've seen it in our setup in 2+ years across 3 clusters. I'd like to see the autoscaler/balancer (or whatever is involved) have improved reliability.
A redeploy works, but before that, the balancer/scaler didn't recognize that this one pod needed to be put on a different node. It was stuck in the above error state for 2+ hours before I saw the issue. (I'm currently in a non-production environment!)
**Are you currently working around this issue?**
A redeploy worked.
**Additional context**
These 3 deployments get deployed together for reasons. So one Gitlab pipeline deploys all 3, one at a time, but in succession as part of the same operation. kubectl is used to create the deployments within kubernetes.
We're currently on platform version eks.12, kubernetes version 1.14 (Yes, we need to update it-- if this was fixed, please let me know what version!)
Contributor guide
Research direction
No repository files, tests, or entry points are named. Start by reviewing the referenced autoscaler and VPC CNI issues, then reproduce or investigate the EKS 1.14 failure involving exhausted node IPs; done means defining and validating autoscaler behavior that places pending pods on another node.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100