bottlerocket-os / bottlerocket-os/bottlerocket

Bug: Max pod and allocatable ram is incorrectly calculated (switch t4g.small from AL2023 to BR, then ram goes from 1437 to 288MB allocatable)

Open
#4,472 7 comments 9 reactions 0 assignees View on GitHub
status/needs-triage type/enhancement
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 18h
Merged PRs (30d)
12

Description

I thought this was a bug at first, but then I RTFM, and saw it was documented behavior, but very unintuitive/unexpected in my opinion.

To use Karpenter you need a MNG with at least 2 baseline nodes.
Originally I was using AL2023 as my baseline nodes, but since bottlerocket is theoretically more secure & recommended I figured I'd switch, but when I did I got pending pods, and when I found out why I was like that's odd. Is this a bug, no I guess not? It's documented as to why. so I figured I'd write it as a feature request. But after looking again, I think it's a bug.

**What I'm using**
* Kube 1.31
* aws-cdk to deploy (using the Layer 2 CDK Construct of eks & MNG, not using Layer 3 construct of EKS Blueprints as I've found it to be buggy and unmaintained.)
* t4g.small (2cpu 2gb ram)
* Bottlerocket AMI = amazon/bottlerocket-aws-k8s-1.31-aarch64-v1.35.0-af533f46
* AL2023 AMI = amazon/amazon-eks-node-al2023-arm64-standard-1.31-v20250403

**Here's what's unintuitive/unexpected**
I switched my baseline nodes from AL2023 to bottlerocket and saw I had pending pods, nodes have insufficient memory. and I was like there's no way these nodes are out of ram, but kubectl describe node said just that.

kubectl describe node against t4g.small (2cpu 2gb ram)
AL2023:
```console
Allocatable:
cpu: 1930m
memory: 1403648Ki <--1437MB (74.2% of node's ram is alloctable to pods)
pods: 110 <-- odd
```

BottleRocket:
```console
Allocatable:
cpu: 1930m
memory: 282052Ki <-- 288MB??? (15% of node's ram is alloctable to pods)
pods: 110 <-- odd
```

The reason the max pods is odd is
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/eni-max-pods.txt
Says t4g.small can support a max of 11 pods.

At first I though this was just a feature request because it seems that allocatable ram usage is documented
https://bottlerocket.dev/en/os/1.31.x/api/settings/kubernetes/#kube-reserved
says `memory_to_reserve = max_num_pods * 11 + 255`
110*11 + 255 = 1465 ram to reserve, well that explains why I see only 288MB ram available.

**But now I think it's a bug**
t4g.small can support a max of 11 pods
so the bug is that the memory_to_reserve is using the wrong value of max_num_pods
it's using 110 pods, when it should be using the instance specific max_num_pods documented in this table https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/eni-max-pods.txt

so bottlerocket should have calculated 11*11 + 255 = 376MB ram to reserve, which would have lead to a value much similar to AL2023.

Contributor guide

Open the contributing guide

Research direction

Start with Bottlerocket's Kubernetes kube-reserved documentation and the eni-max-pods.txt table, then reproduce the result with kubectl describe node on a t4g.small using the listed AMIs. Trace how max_num_pods becomes memory_to_reserve; done means the instance-specific value 11 is used and allocatable memory matches the expected calculation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.