dmlc / dmlc/dgl

[Bug] Labor example cpu sampling & gpu inferencing

Open
#6,221 8 comments 0 reactions 4 assignees Claimed by @Rhett-Ying View on GitHub
bug:confirmed has workaround
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

Run the labor example, utilizing CPU sampling and GPU training. After training, a crash occurs during the inferencing case to calculate final test accuracy.

## To Reproduce

Steps to reproduce the behavior:

`cd examples/pytorch/labor`
`python3 examples/pytorch/labor/train_lightning.py --dataset ogbn-arxiv --num-epochs 1 --data-cpu`

```
Evaluating model in ./tb_logs/ogbn-arxiv_labor_0_False_1/version_43
5%|██████▊ | 2/42 [00:00<00:03, 12.69it/s]
Traceback (most recent call last):
File "/home/ubuntu/workspace/dgl_2/examples/pytorch/labor/train_lightning.py", line 520, in
pred = model.module.inference(
File "/home/ubuntu/workspace/dgl_2/examples/pytorch/labor/model.py", line 81, in inference
h = layer(blocks[0], x)
File "/home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ubuntu/workspace/dgl_2/python/dgl/nn/pytorch/conv/sageconv.py", line 237, in forward
graph.update_all(msg_fn, fn.mean("m", "neigh"))
File "/home/ubuntu/workspace/dgl_2/python/dgl/heterograph.py", line 5110, in update_all
ndata = core.message_passing(
File "/home/ubuntu/workspace/dgl_2/python/dgl/core.py", line 398, in message_passing
ndata = invoke_gspmm(g, mfunc, rfunc)
File "/home/ubuntu/workspace/dgl_2/python/dgl/core.py", line 368, in invoke_gspmm
z = op(graph, x)
File "/home/ubuntu/workspace/dgl_2/python/dgl/ops/spmm.py", line 215, in func
return gspmm(g, "copy_lhs", reduce_op, x, None)
File "/home/ubuntu/workspace/dgl_2/python/dgl/ops/spmm.py", line 111, in gspmm
deg = g.in_degrees()
File "/home/ubuntu/workspace/dgl_2/python/dgl/heterograph.py", line 3669, in in_degrees
v = self.dstnodes(dsttype)
File "/home/ubuntu/workspace/dgl_2/python/dgl/view.py", line 49, in __call__
ret = F.arange(
File "/home/ubuntu/workspace/dgl_2/python/dgl/backend/pytorch/tensor.py", line 407, in arange
return th.arange(start, stop, dtype=dtype, device=ctx)
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f33ce7904d7 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f33ce75a36b in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f33ce82cb58 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x1c36b (0x7f33ce7fd36b in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x2b930 (0x7f33ce80c930 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x4d5a16 (0x7f34256e4a16 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x3ee77 (0x7f33ce775e77 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x1be (0x7f33ce76e69e in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f33ce76e7b9 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: + 0x75afc8 (0x7f3425969fc8 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x305 (0x7f342596a355 in /home/ubuntu/anaconda3/envs/dgl-dev-gpu-dgl-2/lib/python3.10/site-packages/torch/lib/libtorch_python.so)

frame #20: __libc_start_main + 0xf3 (0x7f3445ac7083 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)
```

## Expected behavior

No crash should happen.

## Environment

- DGL Version (e.g., 1.0):
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):
- OS (e.g., Linux):
- How you installed DGL (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information:

## Additional context

More information in: https://github.com/dmlc/dgl/pull/6148

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.