bottlerocket-os / bottlerocket-os/bottlerocket
JVM denied reading /sys/fs/cgroup/memory.max from inside containers on 1.57+ (SELinux container_t)
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
**Image I'm using:**
Bottlerocket OS 1.57.0 and 1.59.0 (`aws-k8s-1.34`), kernel 6.12.73 and 6.12.79.
**What I expected to happen:**
A Java 17 / 21 container with a Kubernetes `memory` limit should be able to read `/sys/fs/cgroup/memory.max` from inside the container so that HotSpot's cgroup-v2 detection can correctly size the heap when `-XX:MaxRAMPercentage` is used.
**What actually happened:**
HotSpot detects cgroup v2 correctly, but the subsequent `open("/sys/fs/cgroup/memory.max")` call fails with `EACCES`. HotSpot then silently falls back to `/proc/meminfo` and sizes the heap against **node** memory, causing `OOMKilled` on every pod whose container limit is less than 50% of node memory.
This is distinct from #4549 (which was resolved in 1.40.0 by enabling `CONFIG_CPUSETS_V1=y`). The `/proc/cgroups` + controller-detection path works correctly on 1.57/1.59 — the new failure is at the subsequent file-read step.
### Minimal reproduction
```yaml
apiVersion: v1
kind: Pod
metadata:
name: corretto-repro
spec:
containers:
- name: corretto
image: amazoncorretto:17.0.19
command: ["java","-Xlog:os+container=trace","-XshowSettings:vm","-version"]
resources:
limits: { cpu: "1", memory: "2Gi" }
requests: { cpu: "1", memory: "2Gi" }
```
Observed output (Corretto 17.0.19, Bottlerocket 1.57.0, kernel 6.12.73, EKS 1.34, `m7a.4xlarge` 64 GiB node):
```
[0.000s][trace][os,container] OSContainer::init: Initializing Container Support
[0.000s][debug][os,container] Detected optional cpuset controller entry in /proc/cgroups
[0.000s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.001s][debug][os,container] Detected cgroups v2 unified hierarchy
[0.001s][trace][os,container] total physical memory: 65972690944
[0.001s][trace][os,container] Path to /memory.max is /sys/fs/cgroup/memory.max
[0.001s][debug][os,container] Open of file /sys/fs/cgroup/memory.max failed, Permission denied
[0.001s][trace][os,container] Memory Limit is: -2
[0.001s][debug][os,container] container memory limit failed: -2, using host value 65972690944
```
From a shell inside the same container:
```
$ id
uid=1001 gid=0(root) groups=0(root)
$ ls -la /sys/fs/cgroup/
ls: cannot open directory '/sys/fs/cgroup/': Permission denied
$ cat /sys/fs/cgroup/memory.max
cat: /sys/fs/cgroup/memory.max: Permission denied
$ cat /proc/self/mountinfo | grep cgroup
... /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,seclabel
$ cat /proc/1/cgroup
0::/
```
### The leaf cgroup is correct; it's just unreadable from `container_t`
From a privileged debug pod on the same node (host-mounted `/sys/fs/cgroup`):
```
leaf: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod.slice/cri-containerd-.scope
memory.max = 8589934592 (correct, matches pod limit)
memory.current = 7005949952
file perms: -rw-r--r-- 1 root root 0 /sys/fs/cgroup/.../memory.max
```
File is POSIX world-readable. Walking up the hierarchy, the only level with a non-`max` `memory.max` is the top-level `kubepods.slice` at node size — the limit is on the leaf, so this is not a parent-cgroup-hierarchy issue (JDK-8322420).
### SELinux contexts
| Actor | Context |
|---|---|
| JVM process (the denied reader) | `system_u:system_r:container_t:s0:c61,c217` |
| Another Java pod on same node | `system_u:system_r:container_t:s0:c895,c961` |
| Privileged debug pod (successful reader) | `system_u:system_r:control_t:s0-s0:c0.c1023` |
| host PID 1 (init) | `system_u:system_r:init_t:s0` |
| kubelet | `system_u:system_r:system_t:s0` |
| containerd | `system_u:system_r:runtime_t:s0` |
When the same `memory.max` file is read from the JVM's mount namespace as UID 1001 but from a context with broader SELinux access (`nsenter -t $JVM_PID -m -S 1001 -G 0 cat /sys/fs/cgroup/memory.max` from the privileged debug pod), the read succeeds. That isolates the variable: file perms are fine, mount is fine, namespace wiring is fine — the denial is in the SELinux policy for `container_t → cgroup_t:file { open read }` (or the equivalent cgroup-v2 class) on 1.57+.
I couldn't pull the exact AVC line because `brush` blocks `journalctl` and `dmesg` from the host (even from a privileged container), and `/var/log/audit` isn't present — so if someone from the team can grab the audit log from a reproducing node, that would nail the missing policy rule.
### Why I believe this is a 1.57+ regression
- Bottlerocket 1.39.x: broken (issue #4549) — different signature, fixed in 1.40.0.
- Bottlerocket 1.40.0 – 1.56.x: presumably working (our older staging nodes were fine; I haven't bisected precisely).
- Bottlerocket 1.57.0 / 1.59.0: `container_t` denied read on cgroup v2 memory files with the trace above.
Community impact: this breaks any workload relying on JVM `-XX:MaxRAMPercentage` (the default recommendation for containerized Java) on Bottlerocket 1.57+. It also likely affects any container that reads its own cgroup files — but JVM is the most visible case because the silent fallback produces immediate OOM kills.
### Workaround
Set absolute `-Xms`/`-Xmx` on every JVM container instead of using percentage flags. This bypasses HotSpot's cgroup detection entirely. Not a long-term fix — it defeats the whole point of container-aware heap sizing.
### Not workarounds (ruled out)
- JDK 17.0.20 with JDK-8349988 backport — file read fails before detection logic runs, so no JDK version helps.
- `enforcing=0` via `settings.boot.kernel-parameters` — works but strips SELinux from every node running JVM workloads. Unacceptable tradeoff.
- Pod-level `seLinuxOptions.type: spc_t` — works per-pod but requires unconfining the container, defeating isolation.
### Ask
- Is this a known SELinux policy regression introduced between 1.56.x and 1.57.0?
- If so, target release for a fix?
- If not, any guidance on grabbing the exact AVC rule that's being denied (given `dmesg`/`journalctl` aren't available from the host control console)?
Happy to test against pre-release AMIs.
Contributor guide
Assessment
This issue has not been assessed yet.