dmlc / dmlc/dgl

[Bug][Dataloader] Device assertion error in multi-gpu runs when num_worker>0

Open
#5,526 9 comments 0 reactions 2 assignees Claimed by @Rhett-Ying View on GitHub
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

In multi-gpu examples, with the current code-base, if we have `num_worker>1` in non-uva mode, DGL would crash with assertion error.

## To Reproduce

1. Slightly modify the example `multigpu/multi_gpu_node_classification.py` for non-uva mode with `num_worker=4`
2. Run `python multi_gpu_node_classification.py --gpu 0,1,2,3`

Error msg and stack trace:
```
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Loss.cu:240: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/pytorch/pytorch/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6c (0x7f8ac03031bc in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xfa (0x7f8ac02c90ea in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3cc (0x7f8ac03912ac in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x16873 (0x7f8ac0360873 in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x249a6 (0x7f8ac036e9a6 in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x4fe99a (0x7f8b044d099a in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so)

```
## Expected behavior

it should run through.

## Environment

- DGL Version (e.g., 1.0): 1.0
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 2.0
- OS (e.g., Linux):
- How you installed DGL (`conda`, `pip`, source): source
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version (if applicable): 12.0
- GPU models and configuration (e.g. V100): A100x4
- Any other relevant information:

## Additional context
I think the assertion error might be due to the excessive overlap of compute stream (default stream) and prefetching stream by checking with profiler.
Also, the crash goes away when `use_alternate_stream=False`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.