bottlerocket-os / bottlerocket-os/bottlerocket
Bug: Max pod and allocatable ram is incorrectly calculated (switch t4g.small from AL2023 to BR, then ram goes from 1437 to 288MB allocatable)
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 12
Description
I thought this was a bug at first, but then I RTFM, and saw it was documented behavior, but very unintuitive/unexpected in my opinion.
To use Karpenter you need a MNG with at least 2 baseline nodes.
Originally I was using AL2023 as my baseline nodes, but since bottlerocket is theoretically more secure & recommended I figured I'd switch, but when I did I got pending pods, and when I found out why I was like that's odd. Is this a bug, no I guess not? It's documented as to why. so I figured I'd write it as a feature request. But after looking again, I think it's a bug.
**What I'm using**
* Kube 1.31
* aws-cdk to deploy (using the Layer 2 CDK Construct of eks & MNG, not using Layer 3 construct of EKS Blueprints as I've found it to be buggy and unmaintained.)
* t4g.small (2cpu 2gb ram)
* Bottlerocket AMI = amazon/bottlerocket-aws-k8s-1.31-aarch64-v1.35.0-af533f46
* AL2023 AMI = amazon/amazon-eks-node-al2023-arm64-standard-1.31-v20250403
**Here's what's unintuitive/unexpected**
I switched my baseline nodes from AL2023 to bottlerocket and saw I had pending pods, nodes have insufficient memory. and I was like there's no way these nodes are out of ram, but kubectl describe node said just that.
kubectl describe node against t4g.small (2cpu 2gb ram)
AL2023:
```console
Allocatable:
cpu: 1930m
memory: 1403648Ki <--1437MB (74.2% of node's ram is alloctable to pods)
pods: 110 <-- odd
```
BottleRocket:
```console
Allocatable:
cpu: 1930m
memory: 282052Ki <-- 288MB??? (15% of node's ram is alloctable to pods)
pods: 110 <-- odd
```
The reason the max pods is odd is
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/eni-max-pods.txt
Says t4g.small can support a max of 11 pods.
At first I though this was just a feature request because it seems that allocatable ram usage is documented
https://bottlerocket.dev/en/os/1.31.x/api/settings/kubernetes/#kube-reserved
says `memory_to_reserve = max_num_pods * 11 + 255`
110*11 + 255 = 1465 ram to reserve, well that explains why I see only 288MB ram available.
**But now I think it's a bug**
t4g.small can support a max of 11 pods
so the bug is that the memory_to_reserve is using the wrong value of max_num_pods
it's using 110 pods, when it should be using the instance specific max_num_pods documented in this table https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/eni-max-pods.txt
so bottlerocket should have calculated 11*11 + 255 = 376MB ram to reserve, which would have lead to a value much similar to AL2023.
Contributor guide
Research direction
Start with Bottlerocket's Kubernetes kube-reserved documentation and the eni-max-pods.txt table, then reproduce the result with kubectl describe node on a t4g.small using the listed AMIs. Trace how max_num_pods becomes memory_to_reserve; done means the instance-specific value 11 is used and allocatable memory matches the expected calculation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100