bottlerocket-os / bottlerocket-os/bottlerocket

Decouple MaxPods from CNI Logic

Open
#1,721 47 comments 26 reactions 0 assignees View on GitHub
area/kubernetes status/research type/enhancement
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**What I'd like:**

I ran into am issue when using https://github.com/awslabs/karpenter and Bottlerocket with the new c6i instance types (see: https://github.com/bottlerocket-os/bottlerocket/issues/1720). Karpenter simulates pod scheduling and provisions instances by discovering instance types from EC2 and binpacking pods onto the node. It uses a formula (14 * AttachableENICount) to compute the max pods value. It also binds the pods before the node comes online as an optimization. If Bottlerocket is unaware of a new instance type, it will default to MaxPods of 110, which is woefully short of the actual number of pods that can be scheduled using the AWS VPC CNI. This causes Karpener's scheduling logic to disagree with the Node's MaxPods, and the Kubelet reports OutOfPods errors for the bound pods.

It's challenging for Bottlerocket to come up with a good default here (i.e. 110), since you have no idea whether or not the unknown instance type is large or small. Calling EC2 Describe Instances on OS boot will introduce significant call volume from large scale ups, as well as increase startup latency.

I see a decent path forward:
1. The latest CNI (with IP Prefix Delegation) support significantly increased max pods per node.
2. Overlay networks like Calico and Cilium support significantly increased max pods per node.
3. Moving forward, I expect that the MaxPods limit will become bound by kubelet cpu/memory overhead instead of AWS VPC limitations.

I'd love to see an option in Bottlerocket to assume that the CNI is not the limiting factor for max pods, and instead compute max pods based off of cpu/memory of the instance (e.g. `max-pods-per-core: 10`), which can cheaply be discovered at boot

**Any alternatives you've considered:**
Looking forward to discussing this path or any others you folks identify.

Contributor guide

Open the contributing guide

Research direction

The issue names no source files or tests; start by reading the 47-comment discussion and the existing MaxPods/CNI configuration paths. Done means an agreed, implemented option to derive MaxPods from CPU and memory rather than CNI limits, with behavior validated for the described Bottlerocket and Karpenter case.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, linux, rust
Domain
networking, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.