aws / aws/containers-roadmap

[EKS] [request]: Exclude pods using the host network from the --max-pods calculation

Open
#2,187 0 comments 3 reactions 0 assignees View on GitHub
EKS EKS Networking Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Exclude pods using the host network from the `--max-pods` calculation.
When calculating `--max-pods`, it is inefficient to fix the number of host network pods at 2.
If using more host network pods other than `aws-node`/`kube-proxy`, available IPs are wasted due to `--max-pods` restrictions.
I expect that when calculating `--max-pods` it will only target pods with private ips.

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When calculating `--max-pods`(https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt), obtain max eni and private ip per eni through the type of node instance and then calculate the total number of ips.
~~~
# Mapping is calculated from AWS EC2 API using the following formula:
# * First IP on each ENI is not used for pods
# * +2 for the pods that use host-networking (AWS CNI and kube-proxy)
#
# # of ENI * (# of IPv4 per ENI - 1) + 2
#
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
~~~
However, as the number of pods using the host network increases, the number of pods using private IPs decreases.
When I use t3.medium type to node instance, which can use 3 ENIs and 6 IPs per ENI. According to [eni-max-pods.txt](https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt), a maximum 17 pods can be placed on this node.

> ignored aws-node warm pool settings

**Case-1 : 2 host-network pods, 15 deployment pods**
~~~
NAMESPACE NAME READY STATUS RESTARTS AGE
game-2048 deployment-2048-764544cd4f-1 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-2 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-3 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-4 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-5 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-6 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-7 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-8 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-9 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-10 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-11 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-12 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-13 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-14 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-15 1/1 Running 0 16m
kube-system aws-node-945nv 1/1 Running 0 16m
kube-system kube-proxy-pf6k5 1/1 Running 0 16m
~~~
~~~
Node ENI/Private IP status
- eni0(6 IPs) : 1 Primay IP + 5 Private IPs
- eni1(6 IPs) : 1 Primay IP + 5 Private IPs
- eni2(6 IPs) : 1 Primay IP + 5 Private IPs
~~~
Can deployed a total of 17 pods, calculated with `--max-pods`, and are using all available IPs.

**Case-2: 4 host-network pods, 13 deployment pods**
~~~
NAMESPACE NAME READY STATUS RESTARTS AGE
game-2048 deployment-2048-764544cd4f-1 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-2 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-3 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-4 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-5 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-6 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-7 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-8 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-9 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-10 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-11 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-12 1/1 Running 0 16m
game-2048 deployment-2048-764544cd4f-13 1/1 Running 0 16m
kube-system aws-node-945nv 1/1 Running 0 16m
kube-system kube-proxy-pf6k5 1/1 Running 0 16m
kube-system efs-csi-controller-6ddf8d6d64-5vbvv 3/3 Running 0 16m
kube-system efs-csi-node-spgfj 3/3 Running 0 16m
~~~
~~~
Node ENI/Private IP status
- eni0(6 IPs) : 1 Primay IP + 5 Private IPs
- eni1(6 IPs) : 1 Primay IP + 5 Private IPs
- eni2(6 IPs) : 1 Primay IP + 3 Private IPs
~~~
Can deployed a total of 17 pods, calculated with `--max-pods`, and left 2 IPs and can't using it.

**Are you currently working around this issue?**
How are you currently solving this problem?

**Additional context**
Anything else we should know?

**Attachments**
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Contributor guide

Open the contributing guide

Research direction

Start with files/eni-max-pods.txt and the issue's EC2 ENI and private-IP formula. Trace how the --max-pods mapping is calculated, then verify that host-network pods beyond aws-node and kube-proxy do not consume private-IP capacity; done means the t3.medium example can use the remaining IPs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.