google / google/gvisor

nvproxy: Support NVIDIA (GH200/GB200) GPUs — missing UVM ioctls and broken ATS initialization

Open
#12,877 7 comments 0 reactions 0 assignees View on GitHub
area: gpu stale-issue type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

Problem 1: MULTI_PROCESS_SHARING_MODE disables mm tracking
gVisor's nvproxy unconditionally forces the UVM_INIT_FLAGS_MULTI_PROCESS_SHARING_MODE flag when initializing the UVM device. This was added as a conservative measure because gVisor's sentry process proxies ioctls on behalf of sandboxed apps, and the original authors thought multi-process sharing mode was needed. I turned
The problem: in the NVIDIA UVM driver, this flag causes uvm_va_space_mm_enabled() to return false ([source: uvm_va_space_mm.c](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/758b4ee8189c5198504cb1c3c5bc29027a9118a3/kernel-open/nvidia-uvm/uvm_va_space_mm.c#L188)), which disables memory management tracking. NVIDIA's own module parameter docs say this: disabling mm tracking "will also disable pageable memory access via either ATS or HMM."
This caused UVM_MM_INITIALIZE to return status 0x10006 (NV_WARN_NOTHING_TO_DO) — the driver saying mm tracking is off, nothing to initialize.

Problem 2: Missing uvm ioctls

Problem 3: Exposing Numa Nodes: https://github.com/google/gvisor/issues/12876

There is a work in progress branch with commits that aim to fix this problem here: https://github.com/google/gvisor/compare/master...luiscape:gvisor:add-missing-arm-ioctl#diff-1186c8000d0492e2995c2423bc81d1b845aa26a1caf94b742cab91ed2bc711a6R193

### Steps to reproduce

Spin up a GH200 instance, install gvisor and run with docker a simple program which tries to initialize the CUDA driver within the container, such as a pytorch test.
Where run config is set in daemon.json to:
`"runsc-gpu": {
"path": "/usr/local/bin/runsc",
"runtimeArgs": [
"--nvproxy=true",
"--nvproxy-allowed-driver-capabilities=compute,utility,video",
"--host-uds=open",
"--debug",
"--debug-log=/tmp/runsc-docker/",
"--strace"
]
}

Also tested with a 64k page kernel on the same ARM GH200 Host, with the exact same errors.
`

### runsc version

```shell

```

### docker version (if using docker)

```shell
Client: Docker Engine - Community
Version: 29.3.1
API version: 1.54
Go version: go1.25.8
Git commit: c2be9cc
Built: Wed Mar 25 16:14:29 2026
OS/Arch: linux/arm64
Context: default

Server: Docker Engine - Community
Engine:
Version: 29.3.1
API version: 1.54 (minimum version 1.40)
Go version: go1.25.8
Git commit: f78c987
Built: Wed Mar 25 16:14:29 2026
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v2.2.2
GitCommit: 301b2dac98f15c27117da5c8af12118a041a31d9
runc:
Version: 1.3.4
GitCommit: v1.3.4-0-gd6d73eb8
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```

### uname

Linux guest 6.8.0-1050-nvidia #53~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 24 15:36:31 UTC 2 aarch64 aarch64 aarch64 GNU/Linux

### kubectl (if using Kubernetes)

```shell

```

### repo state (if built from source)

_No response_

### runsc debug logs (if available)

```shell
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:7977:D0403 14:28:47.647040 1 uvm.go:132] [ 1: 1] nvproxy: uvm ioctl 805306369 = 0x30000001
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:7980:D0403 14:28:47.647103 1 uvm.go:132] [ 1: 1] nvproxy: uvm ioctl 75 = 0x4b
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:7981:D0403 14:28:47.647111 1 uvm_unsafe.go:34] [ 1: 1] nvproxy: uvm ioctl failed: status=0x10006
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:7988:D0403 14:28:47.647159 1 uvm.go:132] [ 1: 1] nvproxy: uvm ioctl 39 = 0x27
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:8615:D0403 14:28:47.666930 1 uvm.go:132] [ 1: 1] nvproxy: uvm ioctl 37 = 0x25
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:8617:D0403 14:28:47.828722 1 task_run.go:319] [ 1: 1] Unhandled user fault: addr=0 ip=e1b67c7fd2f4 access=r-- sig=11 err=bad address
/tmp/runsc-docker/runsc.log.20260403-142847.305539.boot.txt:8908:D0403 14:28:47.829237 1 task_signals.go:200] [ 1: 1] Signal 11, PID: 1, TID: 1, fault addr: 0x0: terminating thread group
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.