vm.swappiness=200 causes page cache to grow too much yet starves pods of their page cache

I hope this issue is clear enough, the problem is pretty complicated (for me) and there is a cascading effect happening.

AI was used to help debugging. I was critical and judgemental of the results from multiple directions.

**Image I'm using:**

- Bottlerocket 1.63.0, `aws-k8s-1.35` variant, on EKS 1.35.
- The nodes are `t3a.medium` instances with 3.8 GiB of memory.
- Each node runs about 15 pods that use 0.5 GiB of RSS in total.
- Kubelet runs with swap disabled for containers and MemoryQoS turned off.

**What I expected to happen:**

When page (filesystem) cache fills the node, the kernel drops cached files that are not in use.

Pods keep their executable pages (program code that is executed constantly) accessible in RAM.

**What actually happened:**

The kernel kept 2.5 GiB of system (filesystem) page cache alive in RAM and didn't want to evict it, because swappiness was 200.

The host has some big journald log files that keep growing over time and so the host ram usage grows with it, it hardly bothers evicting the log files due to this swappiness level. The containerd image file also stays in memory a lot.

To keep room for anything else, it swapped heaps of running processes instead, at like 15MB/s, going to the zram.

At swappiness 200 file lists have weight zero in the kernel. The swap memory relaim scan also happens for the host and kubernetes processes _independently_, because the reclaim scan happens per cgroup.

But the thing is that kubelet blocks swap via their cgroups in my AWS setup, so their 433 MiB of heap stayed resident and can't be evicted.

However, this did not save those pods from problems, because working memory (heap) is not the same as application code (the cached binary code).

This binary code is executed continously because these pods are performing work, but the kernel seems to choose to completely _drop_ the execution code from memory instead (because swap is disabled under kubelet, it doesn't skip the scan.. but does a fallback to drop filesystem cache instead, _despite_ swappiness 200).

So these applications had 3MB of file cache left (which includes their actual code and dependencies) and while executing their code was constantly being read from disk instead, at around 2MB/s.

These k8s pods seem to get pushed away constantly by the log files hardly getting freed from memory (and the host memory growing), and while their working memory is protected by kubernetes, their binary code gets pushed away despite file pages being supposed to survive the most.

An OOMKilled never happened. Instead, pods kept randomly getting killed by liveness probes because they stopped responding quickly enough due to binary code thrashing. They became too slow for kubernetes to consider them healthy.

Kubernetes doesn't notice the memory issue. Kubelet only evicts when `memory.available` drops too low, but about 1.1 GiB still looked available, made of inactive cache and kernel memory that kubelet does not count. The heap that was swapped into zram does not get counted it looks like. Kubelet has no way to know thrashing is happening in the background.

Memory layout on the node, out of 3844 MiB:

| Owner | Resident |
|---|---|
| containerd.service, file cache | 1670 MiB |
| systemd-journald.service, file cache | 835 MiB |
| all pods, file cache | 3 MiB |
| all pods, anonymous (heap/stack/variables/buffers/etc) | 433 MiB |
| kubelet and containerd heap | 3 MiB resident, 272 MiB in zram |

The journald cache is what grows over time. On a quiet node it reaches 2 GiB after 29 days. On a busier node, log will fill faster, and hardly gets evicted from RAM, consuming working memory. We had a pod that was in a crash looping for a while, causing more logs.

As far as I understand it, `vm.swappiness` is the balance between reclaiming application memory (via swap) and reclaiming file cache, where "file cache" includes program code.

At 200 the kernel tries to free application memory almost exclusively wherever swap is allowed, so the host's file cache is barely touched and persists in RAM causing memory use to grow unbalanced on the host side (issue 1)

The intent was probably to make `zram` be more actively used, thus hitting the filesystem less and swapping to compressed memory.

But this may have missed that reclaim is done per cgroup. Pods have swap turned off by kubelet, so inside their cgroup there is no application memory to free, so kernel drops their file cache instead in some way (issue 2). So binary is being read from disk constantly which causes thrashing.

At least, I think that's what causes it? _It's really confusing to see swappiness at 200 (keep binaries alive as much as possible) yet the pod has almost no binary code left in memory._

But the kernel code seems to support the hypothesis (pure AI output here):

> Why the kernel drops pod code instead of skipping the pod, from the kernel source (`mm/vmscan.c`, `get_scan_count()`, v6.12):
> https://github.com/torvalds/linux/blob/v6.12/mm/vmscan.c
>
> ```c
> /* If we have no swap space, do not bother scanning anon folios. */
> if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id, sc)) {
> scan_balance = SCAN_FILE;
> goto out;
> }
> ```
>
> `can_reclaim_anon_pages()` is false for a cgroup with `memory.swap.max=0`, which kubelet sets on every container. `SCAN_FILE` means the reclaim pass for that cgroup scans file cache only; there is no "skip" outcome. A cgroup that is scanned and cannot swap pays in file pages: host cgroups pay in heap (to zram), containers pay in file pages, and for a container that is its own program code.
>
> `memory.swap.max=0` tells the kernel what it may not do with a container's memory, not what it must keep. Without `memory.low`, a container's file cache is as reclaimable as a log file, and it is the only reclaimable thing the container has.
>
> For the host cgroups, which can swap, the file weight at swappiness 200 is `MAX_SWAPPINESS - swappiness = 0`, so their cache is left alone.

**Lowering swappiness fixes thrashing**

Changing `vm.swappiness` from `200` to `60` on a node (``apiclient set --json '{"kernel":{"sysctl":{"vm.swappiness":"60"}}}'`):

| 60-second window | swappiness 200 | swappiness 60 |
|---|---|---|
| pages swapped out per minute | 240,659 | 0 |
| pod file cache resident | 5 MiB | 101 MiB, rising |
| pod code refaults per minute | 28,710 | 2,163 |
| kubelet heap resident | 2 MiB | 82 MiB |
| system cache | flat | shrinking 31 MiB per minute |
| PSI memory, 10-second average | 3.5 to 7 % | 0.00 % |

Same node about 90 minutes after the change:

| Owner | Resident |
|---|---|
| containerd.service, file cache | 1355 MiB |
| systemd-journald.service, file cache | 703 MiB |
| all pods, file cache | 133 MiB |
| all pods, anonymous (heap/stack/variables/buffers/etc) | 511 MiB |
| kubelet and containerd heap | 230 MiB resident, 45 MiB in zram |

Fixes the thrashing, it evicted log pages that the system does not need from host cgroup, stops trying to chew away at the only reclaimable thing the pod cgroup allowed, their code (dropping it almost entirely because binary code can not swap as it's already considered a file by the kernel).

Given all this information, the main issue seems to be that the current swappiness setting creates too much unbalance as it pushes the kernel weight for file pages to zero. But this value has been high for years, I shouldn't have been the first to experience issues. But lowering its value solved the problem and there is not many other knobs I can find.

I believe this could also be a contributing factor behind #4903 (eviction blind spot) and #4916 (kswapd0 CPU).

Start by reproducing the reported Bottlerocket setup with vm.swappiness values of 200 and 60, measuring page refaults, PSI, swap, and pod file cache. Read mm/vmscan.c, especially get_scan_count() and can_reclaim_anon_pages(), alongside the issue's cgroup and kubelet details. Done means confirming or rejecting the proposed reclaim behavior and identifying a scoped change or follow-up supported by measurements.

Written by the indexing model from the issue text.