google / google/gvisor

In-place memory resize doesn't update the guest memory total (MemTotal stays at the boot value)

Open
#14,409 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

I hit this using gVisor pods with Kubernetes in-place pod resize, the same way as #14141 but
for memory. When I raise a pod's memory limit at runtime, the host cgroup is updated
correctly, but the guest keeps reporting the memory it had at boot, so anything that sizes
itself from what the guest reports stays capped at the old value and never uses the memory I
just added, unless I recreate the pod.

Two separate guest-visible signals are stale here, and they matter to different consumers:

- `/proc/meminfo` `MemTotal` (and `sysinfo(2)`), read by `free`-style tooling and by anything
sizing itself from total system memory.
- The sandbox's own cgroup limit, `/sys/fs/cgroup/.../memory.limit_in_bytes`, which is seeded
once when the sandboxed cgroupfs is mounted. Container-aware runtimes prefer this over
`MemTotal` — the JVM's `UseContainerSupport` and .NET's GC heap limit read the cgroup, and
I believe Node's container heuristics do too, though I am less sure of that one.

I originally wrote this issue as though `MemTotal` were the signal those runtimes read. That
was wrong, and it matters for scoping: updating `MemTotal` alone would leave the JVM case
unfixed, and would also make the two signals disagree where today they are at least
consistently stale.

As far as I can tell this is the memory counterpart of #14141: `runsc update` already accepts
`--memory` and the host cgroup does change, but nothing tells the running sentry, so
`usage.MaximumTotalMemoryBytes` is still whatever was computed at sandbox boot. Note #14141's
fix has the same shape on the CPU side — it updates the reported CPU count and leaves the
guest's `cpu.cfs_quota_us` stale — so I would value a maintainer's steer on whether the
cgroup view is meant to be in scope for either.

Environment:

runsc version release-20260817.0 (spec 1.2.1), systrap platform
kind v0.31.0, Kubernetes v1.35.0 (in-place resize is GA), containerd 2.2.0
node image kindest/node:v1.35.0
node: Linux 6.12.76-linuxkit aarch64

Steps to reproduce. Boot a gVisor pod with a 128Mi memory limit, then resize it to 2Gi in
place (no restart):

```yaml
apiVersion: v1
kind: Pod
metadata:
name: gv-mem
spec:
runtimeClassName: gvisor
containers:
- name: c
image: busybox
command: ["sleep","3600"]
resizePolicy:
- resourceName: memory
restartPolicy: NotRequired
resources:
requests: {cpu: "500m", memory: "128Mi"}
limits: {cpu: "500m", memory: "128Mi"}
```

Booted at 128Mi:

```
$ kubectl exec gv-mem -- grep MemTotal /proc/meminfo
MemTotal: 131072 kB
$ kubectl exec gv-mem -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes
134217728
```

Resize in place:

```
$ kubectl patch pod gv-mem --subresource resize --patch \
'{"spec":{"containers":[{"name":"c","resources":{"requests":{"memory":"2Gi"},"limits":{"memory":"2Gi"}}}]}}'
pod/gv-mem patched
```

Kubernetes applied it, and the container was not restarted:

```
spec limit: 2Gi
status alloc: 2Gi
restartCount: 0
```

The host cgroup was updated correctly:

```
$ cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pode71e1fcd_....slice/memory.max
2147483648
```

But the guest still reports the boot-time value, and so does the guest's own cgroup view:

```
$ kubectl exec gv-mem -- grep MemTotal /proc/meminfo
MemTotal: 131072 kB
$ kubectl exec gv-mem -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes
134217728
```

Expected `MemTotal` to follow the new limit (~2 GiB), the way `nproc` is expected to follow a
CPU resize in #14141.

The memory is genuinely usable — only the reported total is stale. Writing 1.5 GiB into tmpfs
inside the same sandbox succeeds while it still reports 128 MiB total:

```
$ kubectl exec gv-mem -- sh -c 'mount -o remount,size=2G /dev/shm; dd if=/dev/zero of=/dev/shm/f bs=1M count=1500'
1572864000 bytes (1.5GB) copied, 2.305989 seconds, 650.5MB/s

$ kubectl exec gv-mem -- df -h /dev/shm
none 3.9G 1.5G 2.4G 38% /dev/shm

$ kubectl exec gv-mem -- grep MemTotal /proc/meminfo
MemTotal: 131072 kB
```

So this is a guest-visibility problem rather than an enforcement one, which makes it the
same shape as #14141: the limit is applied, the guest just cannot see it.

Pointer to the relevant code, in case it helps: `runsc/boot/loader.go` sets both bounds once
at boot,

```go
usage.MinimumTotalMemoryBytes = args.TotalMem
usage.MaximumTotalMemoryBytes = args.TotalMem
```

and `usage.TotalMemory()` clamps to `MaximumTotalMemoryBytes`, which is what
`/proc/meminfo` renders. Nothing updates that ceiling after boot, so it looks analogous to
`ApplicationCores` before #14277.

Happy to work on a patch along the same lines as #14277 (an updater wired through the
existing `runsc update --memory` path) if that direction seems right. Two things I would want
guidance on:

- Whether this should be grow-only, as #14277 is. Lowering the reported total below what is
already allocated would make `MemFree` meaningless — `TotalMemory()` already has a special
case for `memSize < used`.
- What to do about `MinimumTotalMemoryBytes`, since boot sets both bounds to the same value.

Contributor guide

Open the contributing guide

Research direction

Start in runsc/boot/loader.go, then trace usage.TotalMemory and the existing runsc update --memory path, comparing the CPU-side updater from #14277. Confirm how the running sentry receives updates and determine the intended handling of growth, lowering, and MinimumTotalMemoryBytes. Done means the relevant guest memory signals reflect an in-place resize without restarting the sandbox.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
devops, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.