google / google/gvisor

NV50_P2P allocation class unimplemented in nvproxy

Open
#9,827 22 comments 0 reactions 0 assignees View on GitHub
area: gpu revived type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

When running multi-GPU training on A100s applications can attempt to use the unimplemented `NV50_P2P` allocation class.

This presents as a `'mapping of buffer object failed'` error.

```python
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'mapping of buffer object failed'
```

I've take a look at the implementation of this allocation class and unfortunately it's non-trivial: https://github.dev/NVIDIA/open-gpu-kernel-modules/blob/4c29105335610933e744f4ab2524ea63fc39edaf/src/common/sdk/nvidia/inc/class/cl503b.h#L57

Opening this as a tracking issue.

### Steps to reproduce

**Dockerfile**

This Dockerfile runs but I wasn't able to use it to reproduce the issue because I couldn't get an on-demand multi-GPU A100 VM in GCP 😓.

```Dockerfile
FROM nvidia/cuda:12.2.0-devel-ubuntu20.04

RUN apt-get update && apt-get install --yes python3 python3-distutils clang wget vim
RUN wget https://bootstrap.pypa.io/get-pip.py
RUN python3 get-pip.py
RUN python3 -m pip install clang~=10.0.1 # must match version of `clang` installed above.
RUN python3 -m pip install --ignore-installed torch torchvision lightning numpy memory_profiler

COPY <

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.