google / google/gvisor

nvproxy needs GPU PCI topology without requiring RDMA devices

Open
#14,407 2 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

In a multi-GPU nvproxy-only sandbox(without --rdmaproxy), nvidia-smi topo -m fails:

```
nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
Failed to run topology matrix
```

Strace shows the relevant failure:
```
openat("/sys/bus/pci/devices/", O_DIRECTORY) = -1 ENOENT
```

The existing RDMA sysfs collector already includes GPU PCI leaves and ancestors for NCCL GPU-to-NIC distance calculation. However, this only runs when an actual /dev/infiniband/uverbsN device is present:
```
RDMAEnabled = conf.RDMAProxy && specHasUverbsDevice(spec)
```

Could GPU PCI topology collection and rendering be decoupled from RDMA, so nvproxy can expose a read-only /sys/devices/pci... and /sys/bus/pci/devices topology independently, while reusing the RDMA snapshot when one is available?

### Steps to reproduce

docker run --rm --gpus=all --runtime=runsc pytorch/pytorch:2.13.0-cuda13.0-cudnn9-runtime nvidia-smi topo -m

### runsc version

```shell

```

### docker version (if using docker)

```shell

```

### uname

_No response_

### kubectl (if using Kubernetes)

```shell

```

### repo state (if built from source)

_No response_

### runsc debug logs (if available)

```shell

```

Contributor guide

Open the contributing guide

Research direction

Start with the existing RDMA sysfs collector and the RDMAEnabled/specHasUverbsDevice decision described in the issue. Reproduce with the provided docker run command, then trace how PCI leaves and ancestors are exposed; done means nvidia-smi topo -m works in an nvproxy-only sandbox while reusing the RDMA snapshot when available.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
infrastructure, operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.