bottlerocket-os / bottlerocket-os/bottlerocket
Decouple MaxPods from CNI Logic
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**What I'd like:**
I ran into am issue when using https://github.com/awslabs/karpenter and Bottlerocket with the new c6i instance types (see: https://github.com/bottlerocket-os/bottlerocket/issues/1720). Karpenter simulates pod scheduling and provisions instances by discovering instance types from EC2 and binpacking pods onto the node. It uses a formula (14 * AttachableENICount) to compute the max pods value. It also binds the pods before the node comes online as an optimization. If Bottlerocket is unaware of a new instance type, it will default to MaxPods of 110, which is woefully short of the actual number of pods that can be scheduled using the AWS VPC CNI. This causes Karpener's scheduling logic to disagree with the Node's MaxPods, and the Kubelet reports OutOfPods errors for the bound pods.
It's challenging for Bottlerocket to come up with a good default here (i.e. 110), since you have no idea whether or not the unknown instance type is large or small. Calling EC2 Describe Instances on OS boot will introduce significant call volume from large scale ups, as well as increase startup latency.
I see a decent path forward:
1. The latest CNI (with IP Prefix Delegation) support significantly increased max pods per node.
2. Overlay networks like Calico and Cilium support significantly increased max pods per node.
3. Moving forward, I expect that the MaxPods limit will become bound by kubelet cpu/memory overhead instead of AWS VPC limitations.
I'd love to see an option in Bottlerocket to assume that the CNI is not the limiting factor for max pods, and instead compute max pods based off of cpu/memory of the instance (e.g. `max-pods-per-core: 10`), which can cheaply be discovered at boot
**Any alternatives you've considered:**
Looking forward to discussing this path or any others you folks identify.
Contributor guide
Research direction
The issue names no source files or tests; start by reading the 47-comment discussion and the existing MaxPods/CNI configuration paths. Done means an agreed, implemented option to derive MaxPods from CPU and memory rather than CNI limits, with behavior validated for the described Bottlerocket and Karpenter case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, linux, rust
- Domain
- networking, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100