bottlerocket-os / bottlerocket-os/bottlerocket
High `kswapd0` CPU under memory pressure on aws-k8s-1.35+
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
## Summary of the issue
The issue mostly happens on small instances, and can present as:
- `kswapd0` becomes the largest CPU consumer. It reached 91.7% CPU in our reproduction.
- The node shows high system CPU and I/O wait. Our reproduction showed 59.2% system CPU and 21.5% I/O wait.
- Kubernetes becomes stuck launching pods, with pods remaining in `ContainerCreating`.
- Pods fail to start or respond promptly, including readiness probe failures.
- The node becomes `NotReady`.
## Workaround
For node groups where Bottlerocket user data can be configured, zram can be disabled during boot with a bootstrap container:
```toml
[settings.bootstrap-containers.configure-noswap]
mode = "always"
essential = true
user-data = "IyEvYmluL2Jhc2gKc2V0IC1ldXgKZXhlYyAxPiYyCgpST09URlM9Ii8uYm90dGxlcm9ja2V0L3Jvb3RmcyIKWlJBTUNUTD0iL3Vzci9zYmluL3pyYW1jdGwiCgojIERvbmF0ZSBvdXIgY29weSBvZiB6cmFtY3RsIGlmIHRoZSBob3N0IGRvZXNuJ3QgaGF2ZSBpdHMgb3duLgppZiBbICEgLXMgIiR7Uk9PVEZTfS8ke1pSQU1DVEx9IiBdIDsgdGhlbgogIGNwIC91c3Ivc2Jpbi96cmFtY3RsICIke1JPT1RGU30iL2xvY2FsCiAgWlJBTUNUTD0iL2xvY2FsL3pyYW1jdGwiCmZpCgpjaHJvb3QgIiR7Uk9PVEZTfSIgc3dhcG9mZiAvZGV2L3pyYW0wCmNocm9vdCAiJHtST09URlN9IiAiJHtaUkFNQ1RMfSIgLXIgL2Rldi96cmFtMApjaHJvb3QgIiR7Uk9PVEZTfSIgbW9kcHJvYmUgLXIgenJhbQo="
```
This runs the following script:
```bash
#!/bin/bash
set -eux
exec 1>&2
ROOTFS="/.bottlerocket/rootfs"
ZRAMCTL="/usr/sbin/zramctl"
if [ ! -s "${ROOTFS}/${ZRAMCTL}" ]; then
cp /usr/sbin/zramctl "${ROOTFS}"/local
ZRAMCTL="/local/zramctl"
fi
chroot "${ROOTFS}" swapoff /dev/zram0
chroot "${ROOTFS}" "${ZRAMCTL}" -r /dev/zram0
chroot "${ROOTFS}" modprobe -r zram
```
In the reproduction above, removing the zram device returned CPU usage to normal while memory usage remained high.
## Related issues
- [Reproduction and bootstrap-container workaround](https://github.com/bottlerocket-os/bottlerocket/issues/4075#issuecomment-3993015376)
- [User report on Bottlerocket 1.54.0 aws-k8s-1.35](https://github.com/bottlerocket-os/bottlerocket/issues/4075#issuecomment-3965971341)
- [Fixed zram size and default sysctl values](https://github.com/bottlerocket-os/bottlerocket/issues/4903)
Contributor guide
Research direction
Start by reproducing the high kswapd0 CPU and I/O wait under memory pressure on a small aws-k8s-1.35+ instance, then inspect the zram behavior described in the workaround script. Compare the reproduction and user reports in issues 4075 and 4903. Done means the underlying cause is identified and a tested fix or clearly documented resolution prevents the node and pods from becoming unresponsive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, bash, kubernetes, linux
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100