bottlerocket-os / bottlerocket-os/bottlerocket

High `kswapd0` CPU under memory pressure on aws-k8s-1.35+

Open
#4,916 0 comments 4 reactions 0 assignees View on GitHub
status/needs-triage type/bug type/enhancement
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

## Summary of the issue

The issue mostly happens on small instances, and can present as:

- `kswapd0` becomes the largest CPU consumer. It reached 91.7% CPU in our reproduction.
- The node shows high system CPU and I/O wait. Our reproduction showed 59.2% system CPU and 21.5% I/O wait.
- Kubernetes becomes stuck launching pods, with pods remaining in `ContainerCreating`.
- Pods fail to start or respond promptly, including readiness probe failures.
- The node becomes `NotReady`.

## Workaround

For node groups where Bottlerocket user data can be configured, zram can be disabled during boot with a bootstrap container:

```toml
[settings.bootstrap-containers.configure-noswap]
mode = "always"
essential = true
user-data = "IyEvYmluL2Jhc2gKc2V0IC1ldXgKZXhlYyAxPiYyCgpST09URlM9Ii8uYm90dGxlcm9ja2V0L3Jvb3RmcyIKWlJBTUNUTD0iL3Vzci9zYmluL3pyYW1jdGwiCgojIERvbmF0ZSBvdXIgY29weSBvZiB6cmFtY3RsIGlmIHRoZSBob3N0IGRvZXNuJ3QgaGF2ZSBpdHMgb3duLgppZiBbICEgLXMgIiR7Uk9PVEZTfS8ke1pSQU1DVEx9IiBdIDsgdGhlbgogIGNwIC91c3Ivc2Jpbi96cmFtY3RsICIke1JPT1RGU30iL2xvY2FsCiAgWlJBTUNUTD0iL2xvY2FsL3pyYW1jdGwiCmZpCgpjaHJvb3QgIiR7Uk9PVEZTfSIgc3dhcG9mZiAvZGV2L3pyYW0wCmNocm9vdCAiJHtST09URlN9IiAiJHtaUkFNQ1RMfSIgLXIgL2Rldi96cmFtMApjaHJvb3QgIiR7Uk9PVEZTfSIgbW9kcHJvYmUgLXIgenJhbQo="
```

This runs the following script:

```bash
#!/bin/bash
set -eux
exec 1>&2

ROOTFS="/.bottlerocket/rootfs"
ZRAMCTL="/usr/sbin/zramctl"

if [ ! -s "${ROOTFS}/${ZRAMCTL}" ]; then
cp /usr/sbin/zramctl "${ROOTFS}"/local
ZRAMCTL="/local/zramctl"
fi

chroot "${ROOTFS}" swapoff /dev/zram0
chroot "${ROOTFS}" "${ZRAMCTL}" -r /dev/zram0
chroot "${ROOTFS}" modprobe -r zram
```

In the reproduction above, removing the zram device returned CPU usage to normal while memory usage remained high.

## Related issues

- [Reproduction and bootstrap-container workaround](https://github.com/bottlerocket-os/bottlerocket/issues/4075#issuecomment-3993015376)
- [User report on Bottlerocket 1.54.0 aws-k8s-1.35](https://github.com/bottlerocket-os/bottlerocket/issues/4075#issuecomment-3965971341)
- [Fixed zram size and default sysctl values](https://github.com/bottlerocket-os/bottlerocket/issues/4903)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the high kswapd0 CPU and I/O wait under memory pressure on a small aws-k8s-1.35+ instance, then inspect the zram behavior described in the workaround script. Compare the reproduction and user reports in issues 4075 and 4903. Done means the underlying cause is identified and a tested fix or clearly documented resolution prevents the node and pods from becoming unresponsive.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, bash, kubernetes, linux
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.