google / google/gvisor

nvproxy is incompatible with the NVIDIA DRA driver: DRA-allocated GPUs are not correctly scoped

Open
#13,621 4 comments 1 reaction 0 assignees View on GitHub
type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

## nvproxy is incompatible with the NVIDIA DRA driver: DRA-allocated GPUs (incl. supported Tesla T4) are not correctly scoped — sandbox sees all host GPUs, or nvproxy nil-panics in `createDeviceFile`

## Summary

gVisor's nvproxy does not correctly support GPUs allocated via the **NVIDIA DRA driver** (`gpu.nvidia.com`, DynamicResourceAllocation + CDI). This is a compatibility gap between nvproxy and the DRA allocation model, not a hardware limitation: the **Tesla T4 is an officially-supported GPU** and works under gVisor with the classic device-plugin path, but when the *same T4* is allocated via DRA, gVisor cannot provide an isolated sandbox for it.

There is no configuration in which a DRA-allocated GPU is correctly scoped under gVisor. Two mutually exclusive failure modes:

- **Mode 1 (over-exposure):** bare `runsc --nvproxy` starts and CUDA runs, but nvproxy exposes **all** host GPUs regardless of the DRA allocation → tenant-isolation breach.
- **Mode 2 (crash / won't start):** with the NVIDIA CDI runtime wrapper (which applies DRA's device scoping), the sandbox fails to create, and where CDI device nodes are injected, nvproxy nil-panics in `createDeviceFile`.

By contrast, **runc handles the identical DRA allocation correctly** — same node, same claim, same `NVIDIA_VISIBLE_DEVICES=void` — and sees only its allocated GPU. So DRA + CDI is working as designed.

**Why this matters:** DRA is becoming the standard GPU provisioning mechanism in Kubernetes. Today, a supported GPU allocated through DRA cannot be run in an isolated gVisor sandbox at all.

## The core incompatibility

The NVIDIA DRA driver scopes a container to its allocated GPU via **CDI**: it injects the allocated device's `deviceNodes`, registers a `createContainer` hook (`/var/lib/kubelet/plugins/gpu.nvidia.com/nvidia-cdi-hook`), and sets `NVIDIA_VISIBLE_DEVICES=void` (meaning "scoping is done by CDI, not the env var"). runc honors this and is correctly restricted to its allocated GPU.

nvproxy does not participate in this model:
- Seeing `NVIDIA_VISIBLE_DEVICES=void`, it exposes **all** host GPUs instead of the allocated one.
- When the CDI device nodes *are* injected (via the CDI runtime wrapper), it nil-panics creating them.

### Steps to reproduce

## Reproduction — Mode 1 (DRA allocation not scoped; over-exposure)

Node with 2× **Tesla T4**. Two `runtimeClassName: runsc` pods (bare runsc, `nvproxy=true`, no CDI wrapper), each with a DRA ResourceClaim for `count: 1`:

```yaml
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
name: gvisor-iso-a
spec:
devices:
requests:
- name: request-gvisor-iso-a
exactly:
allocationMode: ExactCount
count: 1
deviceClassName: gpu.nvidia.com
selectors:
- cel:
expression: device.attributes['gpu.nvidia.com'].productName == 'Tesla T4'
---
apiVersion: v1
kind: Pod
metadata:
name: gvisor-iso-a
spec:
runtimeClassName: runsc
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: <2-gpu-node>
containers:
- name: cuda
image: nvidia/cuda:12.4.1-runtime-ubuntu22.04
command: ["bash","-c","nvidia-smi -L; sleep 3600"]
resources:
claims:
- name: gpu
resourceClaims:
- name: gpu
resourceClaimName: gvisor-iso-a
```

(Second pod `gvisor-iso-b` identical, with its own claim.)

**Observed — each claim allocated a distinct GPU, but both sandboxes see both:**

gvisor-iso-a -> gpu-1
gvisor-iso-b -> gpu-0
```text
$ kubectl exec gvisor-iso-a -- nvidia-smi -L
GPU 0: Tesla T4 (UUID: GPU-48fdb770-...)
GPU 1: Tesla T4 (UUID: GPU-a5d9d313-...)
$ kubectl exec gvisor-iso-b -- nvidia-smi -L
GPU 0: Tesla T4 (UUID: GPU-48fdb770-...)
GPU 1: Tesla T4 (UUID: GPU-a5d9d313-...)
$ kubectl exec gvisor-iso-a -- printenv NVIDIA_VISIBLE_DEVICES
void
```
**Control (runc, same node, same DRA claim, same `void`):** sees **only** its allocated GPU (one UUID). DRA/CDI scoping works for runc; nvproxy ignores it.

**Attempts to force scoping that did NOT work:**
- Hand-editing the generated CDI file (`/var/run/cdi/k8s.gpu.nvidia.com-claim_*.yaml`) `void`→`` → regenerated back to `void` by the DRA driver on pod create.

The generated per-claim CDI spec contains `NVIDIA_VISIBLE_DEVICES=void`, a `createContainer` hook, and `deviceNodes` for the allocated GPU plus `/dev/nvidia-modeset`, `/dev/nvidia-caps/nvidia-cap1`, `/dev/nvidia-caps/nvidia-cap2`.

## Reproduction — Mode 2 (CDI wrapper: crash / won't start)

Runsc handler configured to apply DRA's CDI scoping via the wrapper:
```text
binary_name = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
```
Result: `OCI runtime create failed: /usr/local/nvidia/toolkit/nvidia-container-runtime.cdi did not terminate successfully` at sandbox creation. With `/etc/nvidia-container-runtime/config.toml` present (`root=/run/nvidia/driver`, `mode=cdi`, `runtimes=["runsc"]`), the wrapper still exits non-zero.

Where CDI device nodes were injected into the runsc spec, nvproxy nil-panicked during sub-container device-file creation:
```text
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: code=0x1 addr=0x10]
gvisor.dev/gvisor/runsc/boot.createDeviceFile
runsc/boot/vfs.go:1609
gvisor.dev/gvisor/runsc/boot.createDeviceFiles
runsc/boot/vfs.go:1519
gvisor.dev/gvisor/runsc/boot.setupContainerVFS
runsc/boot/vfs.go:227
gvisor.dev/gvisor/runsc/boot.(*Loader).createContainerProcess
runsc/boot/loader.go:1340
gvisor.dev/gvisor/runsc/boot.(*Loader).startSubcontainer
runsc/boot/loader.go:1258
gvisor.dev/gvisor/runsc/boot.(*containerManager).StartSubcontainer
runsc/boot/controller.go:496
````
Key frames:
```text
panic: runtime error: invalid memory address or nil pointer dereference [SIGSEGV]
gvisor.dev/gvisor/runsc/boot.createDeviceFile runsc/boot/vfs.go:1609
gvisor.dev/gvisor/runsc/boot.createDeviceFiles runsc/boot/vfs.go:1519
gvisor.dev/gvisor/runsc/boot.setupContainerVFS runsc/boot/vfs.go:227
...(*Loader).createContainerProcess runsc/boot/loader.go:1340
...(*Loader).startSubcontainer runsc/boot/loader.go:1258
...(*containerManager).StartSubcontainer runsc/boot/controller.go:496
```
Surfaces to kubelet as starting sub-container [...]: StartSubcontainer failed: EOF. Suspected trigger: a DRA-injected device node (/dev/nvidia-caps/* and/or /dev/nvidia-modeset) — a classic device-plugin GPU pod (which lacks these) does not crash.
#### Expected behavior
A DRA-allocated GPU (e.g. Tesla T4) should be usable in an isolated gVisor sandbox, scoped to exactly the allocated device — matching runc. nvidia-smi -L inside the sandbox should list only the allocated GPU(s). nvproxy should honor the DRA/CDI scoping (or otherwise restrict to the allocated device), and must not nil-panic on the DRA-injected device set, nor default to exposing all host GPUs on NVIDIA_VISIBLE_DEVICES=void.
#### Questions for maintainers

- Is the NVIDIA DRA driver supported with nvproxy today? If so, what is the expected way to get an isolated, correctly-scoped DRA-allocated GPU into a gVisor sandbox?
- Should nvproxy honor the DRA/CDI createContainer hook + deviceNodes scoping (as runc does), rather than reading NVIDIA_VISIBLE_DEVICES alone?
- Are /dev/nvidia-caps/* and /dev/nvidia-modeset supported by nvproxy's createDeviceFile? (nil-panic at vfs.go:1609)

## Environment

- **runsc version:** `release-20260622.0` (reproduced on `release-20260525.0`, `release-20260601.0`)
- **spec:** 1.2.1 · **Platform:** systrap
- **containerd:** `v2.2.2`, config version 3 (`io.containerd.cri.v1.runtime`)
- **Kubernetes:** `v1.35.5`
- **Node OS / kernel:** `Ubuntu 24.04.3 LTS / 6.8.0-124-generic`
- **GPUs:** **Tesla T4 (Turing, officially supported)** Host healthy (`nvidia-smi` clean).
- **NVIDIA driver:** 580.126.20 (`runsc nvproxy list-supported-driver`), open kernel modules
- **GPU provisioning:** NVIDIA GPU Operator driver-container (driver at `/run/nvidia/driver`) + **NVIDIA DRA driver** (`nvidia-dra-driver-gpu`, `gpu.nvidia.com`) version `0.4.1`; device plugin disabled; `cdi.enabled=true`
- **nvidia-container-toolkit:** 1.19.0
- **runsc handler:** `nvproxy=true`, `debug=true`, `debug-log`, `strace=true`

Node prep for the GPU-Operator driver-container layout (may be relevant): `nvidia-container-cli` symlinked onto PATH; `/etc/ld.so.conf.d/*.conf` = `/run/nvidia/driver/usr/lib/x86_64-linux-gnu` + `ldconfig`, so nvproxy's `nvidia-container-cli` resolves driver libs.

### uname
```shell
Linux stage-wrkld-worker-7 6.8.0-124-generic #124-Ubuntu SMP PREEMPT_DYNAMIC Tue May 26 13:00:45 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
```
### kubectl (if using Kubernetes)

```shell
kubectl version
Client Version: v1.35.5
Kustomize Version: v5.7.1
Server Version: v1.35.5

kubectl get nodes
NAME STATUS ROLES AGE VERSION
stage-wrkld-worker-4 Ready tesla-t4 212d v1.35.5
stage-wrkld-worker-5 Ready tesla-t4 192d v1.35.5
stage-wrkld-worker-7 Ready tesla-t4 212d v1.35.5
stg-nc-wkld-master-1 Ready control-plane 215d v1.35.5
stg-nc-wkld-master-2 Ready control-plane 215d v1.35.5
stg-nc-wkld-master-3 Ready control-plane 215d v1.35.5
stg-nc-wkld-worker-1 Ready 215d v1.35.5
stg-nc-wkld-worker-2 Ready 215d v1.35.5
stg-nc-wkld-worker-3 Ready 215d v1.35.5
stg-nc-wkld-worker-8 Ready 42d v1.35.5
stg-nc-wkld-worker-9 Ready 42d v1.35.5
```
### runsc debug logs (if available)

```shell
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x10245bb]

goroutine 306 gp=0xdf4eed1c000 m=14 mp=0xdf4ee569808 [running]:
panic({0x1278340?, 0x6f195b0?})
GOROOT/src/runtime/panic.go:879 +0x16f fp=0xdf4eee924c8 sp=0xdf4eee92418 pc=0x48c52f
runtime.panicmem(...)
GOROOT/src/runtime/panic.go:336
runtime.sigpanic()
GOROOT/src/runtime/signal_unix.go:931 +0x378 fp=0xdf4eee92528 sp=0xdf4eee924c8 pc=0x48f378
gvisor.dev/gvisor/runsc/boot.createDeviceFile({0x1548f20, 0xdf4ee9e78c0}, 0xdf4eec1c420, 0xdf4ee6b2380, 0xdf4ee9001e0, {0xdf4ee5cad10?, 0xdf4ee8d8a08?}, {{0xdf4eee504e0, 0xf}, {0x1565878, ...}, ...})
runsc/boot/vfs.go:1609 +0x71b fp=0xdf4eee92630 sp=0xdf4eee92528 pc=0x10245bb
gvisor.dev/gvisor/runsc/boot.createDeviceFiles({0x1548f20, 0xdf4ee9e78c0}, 0xdf4eec1c420, 0xdf4ee6b2380, 0xdf4ee9001e0, {0xdf4ee5cad10?, 0xdf4ee8d8a08?})
runsc/boot/vfs.go:1519 +0x93a fp=0xdf4eee92888 sp=0xdf4eee92630 pc=0x1023e1a
gvisor.dev/gvisor/runsc/boot.setupContainerVFS({0x1548f20, 0xdf4ee9e76a0}, 0xdf4ee6b2380, 0xdf4ee5ca9a0, 0xdf4ee6b23b0)
runsc/boot/vfs.go:227 +0x42e fp=0xdf4eee92a30 sp=0xdf4eee92888 pc=0x101602e
gvisor.dev/gvisor/runsc/boot.(*Loader).createContainerProcess(0xdf4ee7e8008, 0xdf4ee6b2380)
runsc/boot/loader.go:1340 +0x6d3 fp=0xdf4eee92d88 sp=0xdf4eee92a30 pc=0x1002b13
gvisor.dev/gvisor/runsc/boot.(*Loader).startSubcontainer(0xdf4ee7e8008, 0xdf4eeafe1b0, 0xdf4eee5e008, {0xdf4ee8c35c0, 0x40}, {0xdf4ee85ab28, 0x3, 0x3}, {0xdf4ee872e00, 0x3f, ...}, ...)
runsc/boot/loader.go:1258 +0xce6 fp=0xdf4eee93288 sp=0xdf4eee92d88 pc=0x1001b66
gvisor.dev/gvisor/runsc/boot.(*containerManager).StartSubcontainer(0xdf4eeaaee40, 0xdf4ee7acaf0, 0x0?)
runsc/boot/controller.go:496 +0xf28 fp=0xdf4eee93608 sp=0xdf4eee93288 pc=0xfed2c8
runtime.call32(0xdf4ee817980, 0xdf4ee83a490, 0x0, 0x0, 0x0, 0x18, 0xdf4eee93870)
src/runtime/asm_amd64.s:836 +0x3a fp=0xdf4eee93638 sp=0xdf4eee93608 pc=0x493d1a
runtime.reflectcall(0x11d1e00?, 0x6f6ec60?, 0x4?, 0x14a7ee0?, 0x0?, 0x12?, 0x11d1e00?)
:1 +0x2d fp=0xdf4eee93678 sp=0xdf4eee93638 pc=0x49794d
reflect.Value.call({0xdf4ee948ba0?, 0xdf4ee83a490?, 0xdf4ee623c18?}, {0x1496b23, 0x4}, {0xdf4ee623ea8, 0x3, 0xdf4ee623c48?})
GOROOT/src/reflect/value.go:586 +0xf0c fp=0xdf4eee93be0 sp=0xdf4eee93678 pc=0x4e79ec
reflect.Value.Call({0xdf4ee948ba0?, 0xdf4ee83a490?, 0x8000?}, {0xdf4ee623ea8?, 0xdf4ee7acaf0?, 0x16?})
GOROOT/src/reflect/value.go:369 +0xb9 fp=0xdf4eee93c58 sp=0xdf4eee93be0 pc=0x4e6899
gvisor.dev/gvisor/pkg/urpc.(*Server).handleOne(0xdf4ee82c320, 0xdf4eed96080)
pkg/urpc/urpc.go:343 +0x69f fp=0xdf4eee93f88 sp=0xdf4eee93c58 pc=0xcc0cdf
gvisor.dev/gvisor/pkg/urpc.(*Server).handleRegistered(...)
pkg/urpc/urpc.go:454
gvisor.dev/gvisor/pkg/urpc.(*Server).StartHandling.func1()
pkg/urpc/urpc.go:474 +0x67 fp=0xdf4eee93fe0 sp=0xdf4eee93f88 pc=0xcc1d47
runtime.goexit({})
src/runtime/asm_amd64.s:1771 +0x1 fp=0xdf4eee93fe8 sp=0xdf4eee93fe0 pc=0x4951e1
created by gvisor.dev/gvisor/pkg/urpc.(*Server).StartHandling in goroutine 102
pkg/urpc/urpc.go:472 +0x6b
```

Contributor guide

Open the contributing guide

Research direction

Start with runsc/boot/vfs.go at createDeviceFile and createDeviceFiles, then follow the setup path through runsc/boot/loader.go and controller.go using the supplied stack trace. Reproduce both DRA/CDI modes and compare the generated device set with the classic device-plugin path. Done means a DRA-allocated GPU starts without a panic and nvidia-smi -L shows only the allocated GPU.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.