Pod CPU/memory limits not enforced on gVisor pods — CRI sandbox-cpu-* annotations not translated into sandbox OCI spec

# Pod CPU/memory limits not enforced on gVisor pods — CRI `sandbox-cpu-*` annotations not translated into sandbox OCI spec

## TL;DR

On a Kubernetes cluster running gVisor (`RuntimeClass: gvisor`) via `containerd-shim-runsc-v1`, **pod-level CPU and memory limits are silently unenforced**. A pod with `limit: cpu=660m` sustains **1675 mCPU (2.54× the limit)** with `nr_throttled = 0`. The host cgroup mechanism works — writing `cpu.max` to the sandbox scope manually enforces immediately — but neither containerd nor the shim populates it for gVisor sandboxes.

Root cause (verified from the OCI spec containerd wrote to disk and from `google/gvisor` master source): the pod sandbox's OCI `config.json` has only `linux.resources.cpu.shares = 2`. The real pod-level CPU limits are present, but only in the CRI annotations `io.kubernetes.cri.sandbox-cpu-{quota,period,shares}` and `sandbox-memory`. `containerd-shim-runsc-v1` does not read those annotations, so runsc receives no quota and writes nothing to `cpu.max` on the sandbox scope.

Per-container OCI specs *are* populated correctly, but under gVisor **all pod containers share one sandbox process** — so per-container cgroup limits never bite the actual workload. The sandbox scope is the only host cgroup where a pod-wide limit could bite, and it's unlimited.

## Environment

- containerd `2.2.4+unknown` (config format `version = 3`, `[plugins.'io.containerd.cri.v1.runtime']`)
- runsc `release-20260216.0`, spec `1.1.0-rc.1`, platform systrap, cgroup v2 host
- Amazon Linux 2023 (kernel 6.12.92), systemd cgroup driver
- Kubernetes 1.34 (Amazon EKS)
- Runtime block:
```toml
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
runtime_type = 'io.containerd.runsc.v1'
sandboxer = '' # defaults to 'podsandbox'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc.options]
ConfigPath = '/etc/containerd/runsc.toml'
TypeUrl = 'io.containerd.runsc.v1.options'
```
- `/etc/containerd/runsc.toml`:
```toml
[runsc_config]
network = "host"
systemd-cgroup = "true"
```

## Reproduction

Deploy any gVisor pod with a CPU limit (here: `request: 608m`, `limit: 660m`).

### Step 1 — sandbox OCI spec

Read `/run/containerd/io.containerd.runtime.v2.task/k8s.io//config.json`:

```jsonc
"linux": {
"cgroupsPath": "kubepods-burstable-pod.slice:cri-containerd:",
"resources": {
"cpu": { "shares": 2 } // ← only shares:2, no quota, no period, no memory
}
},
"annotations": {
"io.kubernetes.cri.container-type": "sandbox",
"io.kubernetes.cri.sandbox-cpu-period": "100000",
"io.kubernetes.cri.sandbox-cpu-quota": "85800", // ← real limits sit here
"io.kubernetes.cri.sandbox-cpu-shares": "684",
"io.kubernetes.cri.sandbox-memory": "2304770048"
}
```

For comparison, the app container's OCI spec in the same pod is populated correctly:

```jsonc
"linux": {
"resources": {
"cpu": { "shares": 622, "quota": 66000, "period": 100000 },
"memory": { "limit": 2147483648, "swap": 2147483648 }
}
}
```

### Step 2 — host cgroup on the sandbox scope

```
$ cat /sys/fs/cgroup/kubepods.slice/.../kubepods-burstable-pod.slice/cri-containerd-.scope/cpu.max
max 100000

$ cat .../cri-containerd-.scope/cpu.weight
1
```

The pod slice itself (parent of the sandbox scope) is *also* unlimited:

```
$ cat kubepods-burstable-pod.slice/cpu.max
max 100000

$ cat kubepods-burstable-pod.slice/cpu.weight
27
```

### Step 3 — CPU stress test, run from inside the sandbox

Spawned 8 pure-CPU-spin Python workers **inside the app container** for 60 s, sampled `cpu.stat` on the sandbox scope before/after:

```python
python3 -c "
import multiprocessing as mp, time
def spin(deadline):
while time.monotonic() < deadline: pass
dl = time.monotonic() + 60
ps = [mp.Process(target=spin, args=(dl,)) for _ in range(8)]
for p in ps: p.start()
for p in ps: p.join()
"
```

`cpu.stat` delta over the 62 s window:

| Field | Delta |
|---|---|
| `usage_usec` | **103,863,066 → 1675 mCPU sustained (2.54× the 660m limit)** |
| `user_usec` | 95,287,038 |
| `system_usec` | 8,576,029 |
| `nr_periods` | **0** |
| `nr_throttled` | **0** |
| `throttled_usec` | **0** |
| `nr_bursts` | 0 |

The kernel CFS never engaged because `cpu.max = max`. On a 2-vCPU node, the sandbox sustained ~84% of the physical node CPU with zero throttling.

### Control — kernel enforcement works when `cpu.max` is set

Manually setting the limit on the sandbox scope:

```
$ echo "66000 100000" > .../cri-containerd-.scope/cpu.max
```

…and re-running the same load produces `nr_throttled = 602` in a 60 s window and CPU drops to the expected ~660 mCPU. **The host mechanism works; only the wiring from containerd/shim into the sandbox spec is missing.**

## Why `systemd-cgroup = "true"` doesn't fix it (re: #12392)

Enabling `--systemd-cgroup=true` correctly moves the sandbox scope under `kubepods.slice/.../*.scope` (out of `runtime.slice`), and the debug log confirms:

```
Config.SystemdCgroup (--systemd-cgroup): true
Installing systemd cgroup resource controller under kubepods-burstable-pod.slice
Joining systemd cgroup cri-containerd-.scope
```

But enforcement still doesn't happen, because the resource-set runsc installs into that scope is still the sandbox spec's `linux.resources` — which contains only `shares:2`. Correct hierarchy, still no `cpu.max`.

## Root cause in upstream source (verified against `master` @ 2026-07-20)

- [`pkg/shim/v1/utils/annotations.go`](https://github.com/google/gvisor/blob/master/pkg/shim/v1/utils/annotations.go) — the shim's entire annotation allowlist is two constants: `io.kubernetes.cri.container-type` and `io.kubernetes.cri.sandbox-log-directory`. No `sandbox-cpu-*` or `sandbox-memory` handling anywhere in `pkg/shim/`.
- [`pkg/shim/v1/runsc/service.go`](https://github.com/google/gvisor/blob/master/pkg/shim/v1/runsc/service.go) — `setPodCgroup` is the shim's only pod-cgroup logic. It parses `spec.Linux.CgroupsPath` and injects `dev.gvisor.spec.cgroup-parent` as an annotation. It does **not** hydrate `spec.Linux.Resources`.
- [`runsc/container/container.go`](https://github.com/google/gvisor/blob/master/runsc/container/container.go) — `setupCgroupForRoot` installs the parent cgroup via `parentCgroup.Install(spec.Linux.Resources)`. Whatever the shim handed it is what runsc applies verbatim.
- [`pkg/shim/v1/runsc/options.go`](https://github.com/google/gvisor/blob/master/pkg/shim/v1/runsc/options.go) — the full `Options` struct has no field that reads CRI sandbox annotations. **There is no config toggle that would enable this.**
- [`pkg/shim/v1/runsc/sandbox.go`](https://github.com/google/gvisor/blob/master/pkg/shim/v1/runsc/sandbox.go) — every method of the shim's `SandboxController` implementation (`CreateSandbox`, `StartSandbox`, `Platform`, `StopSandbox`, `WaitSandbox`, `SandboxStatus`, `PingSandbox`, `ShutdownSandbox`, `SandboxMetrics`) returns `errdefs.ErrNotImplemented`. So setting `sandboxer = "shim"` on the runtime is not a workaround — it would fail outright.

## Expected behavior

For a pod-sandbox created via containerd's CRI plugin (`sandboxer = "podsandbox"`), when the sandbox OCI spec contains the CRI annotations `io.kubernetes.cri.sandbox-cpu-{quota,period,shares}` and `io.kubernetes.cri.sandbox-memory`, `containerd-shim-runsc-v1` should either:

1. **Preferred:** hydrate `spec.linux.resources.cpu` (`quota`, `period`, `shares`) and `spec.linux.resources.memory.limit` from those annotations before invoking runsc, so runsc's `cgroupInstall` writes the correct `cpu.max` / `cpu.weight` / `memory.max` on the sandbox scope; or
2. **Alternative:** write those limits directly to the sandbox host cgroup after runsc creates the scope.

Either approach fixes the observed behavior.

## Impact

- Silent — no error, no log line indicating enforcement is off.
- Affects every gVisor pod on every cluster whose shim was built from a commit where the annotation → spec translation is absent. Based on master @ 2026-07-20, that's every current build.
- Multi-tenant clusters relying on gVisor for both isolation *and* fairness silently lose fairness. A misbehaving tenant can consume the whole node.
- Memory limits are affected too: `sandbox-memory` annotation is present, sandbox spec has no `memory.limit`, so gVisor won't OOM-kill at the pod limit either.

Happy to test a proposed fix on this same cluster and report back with the same before/after evidence.

Start with pkg/shim/v1/utils/annotations.go and pkg/shim/v1/runsc/service.go, then trace runsc/container/container.go's setupCgroupForRoot. Use the provided sandbox OCI and cgroup checks to verify how CRI annotations reach Linux resources; done means CPU and memory limits are enforced on the gVisor sandbox scope for CRI pods.

Written by the indexing model from the issue text.