google / google/gvisor

nvproxy: Support NVIDIA driver 595.71.05

Open Beginner friendly
#14,617 2 comments 0 reactions 0 assignees View on GitHub
type: enhancement
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

### Request

Please add NVIDIA driver `595.71.05` (x86_64) to nvproxy's qualified driver set.

### Motivation

I'm an engineer at Vast.ai, levaluating gVisor as the container runtime for renter workloads on
our hosts. `595.71.05` is one of the most widely deployed driver versions I see across the
fleet, and no 595 version currently appears in `runsc nvproxy list-supported-drivers`, so that
hardware can't run under gVisor without `--nvproxy-allow-unsupported-driver`.

### What I tested

`595.71.05` is already present in nvproxy's ABI tree, it runs with
`--nvproxy-allow-unsupported-driver` rather than being rejected as unknown. I ran a validation
pass in case it helps toward qualification.

- **runsc:** `release-20260831.0` (spec 1.2.1)
- **Driver:** 595.71.05
- **Hardware:** 4× NVIDIA GeForce RTX 3090, x86_64, 28 CPUs
- **Image:** `pytorch/pytorch` CUDA runtime
- **Invocation:** `runsc --nvproxy --nvproxy-allow-unsupported-driver`, via
nvidia-container-runtime in `legacy` mode

**Correctness (all pass):**

| check | result |
|---|---|
| `torch.cuda.is_available()` / `device_count` | true / 4 |
| Per-device fp32 matmul vs CPU reference (`allclose`, rtol/atol 1e-3) | correct on all 4 devices |
| Host↔device int32 round-trip, 2^20 elements (`torch.equal`) | correct |
| Cross-device copy `cuda:0` → `cuda:1`, verified | correct |
| Deterministic reduction over 2^22 fp64 elements | correct |
| Distinct constant per device, summed independently | correct (no aliasing) |

`can_device_access_peer(0,1)` reports `false`, but so does the same query on bare metal, these
are consumer GeForce cards without an NVLink bridge, so direct P2P is unavailable in hardware
rather than being an nvproxy gap.

**Real workload — no missing ioctl handlers:**

50 training iterations of Conv2d → BatchNorm2d → ReLU → Conv2d → AdaptiveAvgPool2d → Linear
with `AdamW` and `cross_entropy`, exercising cuDNN, autograd and optimizer state. Completes
normally, final loss at chance level for 10 classes on random input, as expected.

```python
import torch, torch.nn as nn
m = nn.Sequential(nn.Conv2d(3,64,3,padding=1), nn.BatchNorm2d(64), nn.ReLU(),
nn.Conv2d(64,128,3,padding=1), nn.AdaptiveAvgPool2d(1), nn.Flatten(),
nn.Linear(128,10)).cuda()
opt = torch.optim.AdamW(m.parameters())
for _ in range(50):
x = torch.randn(64,3,64,64, device="cuda"); y = torch.randint(0,10,(64,), device="cuda")
loss = nn.functional.cross_entropy(m(x), y); loss.backward(); opt.step(); opt.zero_grad()
torch.cuda.synchronize(); print("final loss", loss.item())
```

With `--debug --debug-log=...`, **zero `nvproxy: handler is undefined` lines** across the run.

### One observation, likely unrelated

I saw four of these during CUDA initialization:

```
I0904 19:48:18.254412 1 object.go:260] [1:1] nvproxy: freeing object with unknown handle 0xc1d00447:0x5c000002
I0904 19:48:18.285585 1 object.go:260] [1:1] nvproxy: freeing object with unknown handle 0xc1d00447:0x5c000009
I0904 19:48:18.306779 1 object.go:260] [1:1] nvproxy: freeing object with unknown handle 0xc1d00447:0x5c000010
I0904 19:48:18.333863 1 object.go:260] [1:1] nvproxy: freeing object with unknown handle 0xc1d00447:0x5c000017
```

The count is 4 regardless of whether 1 or 4 GPUs are visible, so it isn't per-device.

**I don't think this is 595-specific.** The same message appears on a *qualified* driver
(`580.173.02`, same runsc build) running the same workload — 1 occurrence there vs 4 here. That
comparison is confounded (different CUDA runtime in the image, different GPU model), so I'm
reporting it for completeness rather than as a defect. Happy to run a controlled comparison if
it would be useful.

### Also of interest

If it's cheap to qualify additional 595 versions alongside this one, `595.84` and `595.58.03`
are also common on our hosts. I haven't validated those yet, and I'm happy to run the same pass
on them if that would help.

### Is this feature related to a specific bug?

_No response_

### Do you have a specific solution in mind?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at nvproxy’s qualified driver set and the `runsc nvproxy list-supported-drivers` entry point, checking how existing ABI-tree driver versions are qualified. Add 595.71.05 to the qualified set, then run the supported-driver listing and the reported CUDA validation workload to confirm it is accepted without the override.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
operating-systems
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.