dmlc / dmlc/dgl

[GraphBolt] Cannot re-initialize CUDA in forked subprocess

Open
#6,981 10 comments 1 reaction 1 assignee Claimed by @mfbalin View on GitHub
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

## To Reproduce

Steps to reproduce the behavior:

1. `python3 examples/sampling/graphbolt/node_classification.py --num-workers 4`

```
File "/opt/conda/envs/dgl-dev-gpu-dgl-0/lib/python3.10/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 183, in wrap_generator
response = gen.send(None)
File "/home/ubuntu/workspace/dgl_0/python/dgl/graphbolt/base.py", line 209, in __iter__
data = recursive_apply(data, apply_to, self.device)
File "/home/ubuntu/workspace/dgl_0/python/dgl/utils/internal.py", line 1135, in recursive_apply
return fn(data, *args, **kwargs)
File "/home/ubuntu/workspace/dgl_0/python/dgl/graphbolt/base.py", line 145, in apply_to
return x.to(device) if hasattr(x, "to") else x
File "/home/ubuntu/workspace/dgl_0/python/dgl/graphbolt/minibatch.py", line 496, in to
setattr(self, attr, apply_to(getattr(self, attr), device))
File "/home/ubuntu/workspace/dgl_0/python/dgl/graphbolt/minibatch.py", line 462, in apply_to
return recursive_apply(x, lambda x: _to(x, device))
File "/home/ubuntu/workspace/dgl_0/python/dgl/utils/internal.py", line 1135, in recursive_apply
return fn(data, *args, **kwargs)
File "/home/ubuntu/workspace/dgl_0/python/dgl/graphbolt/minibatch.py", line 462, in
return recursive_apply(x, lambda x: _to(x, device))
File "/home/ubuntu/workspace/dgl_0/python/dgl/graphbolt/minibatch.py", line 459, in _to
return x.to(device) if hasattr(x, "to") else x
File "/opt/conda/envs/dgl-dev-gpu-dgl-0/lib/python3.10/site-packages/torch/cuda/__init__.py", line 284, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
This exception is thrown by __iter__ of CopyTo(datapipe=ShardingFilterIterDataPipe, device=device(type='cuda'), extra_attrs=['seed_nodes'])
```

## Expected behavior

## Environment

- DGL Version (e.g., 1.0): master
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):
- OS (e.g., Linux):
- How you installed DGL (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information:

## Additional context

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.