NV50_P2P allocation class unimplemented in nvproxy
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
When running multi-GPU training on A100s applications can attempt to use the unimplemented `NV50_P2P` allocation class.
This presents as a `'mapping of buffer object failed'` error.
```python
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'mapping of buffer object failed'
```
I've take a look at the implementation of this allocation class and unfortunately it's non-trivial: https://github.dev/NVIDIA/open-gpu-kernel-modules/blob/4c29105335610933e744f4ab2524ea63fc39edaf/src/common/sdk/nvidia/inc/class/cl503b.h#L57
Opening this as a tracking issue.
### Steps to reproduce
**Dockerfile**
This Dockerfile runs but I wasn't able to use it to reproduce the issue because I couldn't get an on-demand multi-GPU A100 VM in GCP 😓.
```Dockerfile
FROM nvidia/cuda:12.2.0-devel-ubuntu20.04
RUN apt-get update && apt-get install --yes python3 python3-distutils clang wget vim
RUN wget https://bootstrap.pypa.io/get-pip.py
RUN python3 get-pip.py
RUN python3 -m pip install clang~=10.0.1 # must match version of `clang` installed above.
RUN python3 -m pip install --ignore-installed torch torchvision lightning numpy memory_profiler
COPY <
Contributor guide
Assessment
This issue has not been assessed yet.