instadeepai / instadeepai/LightMHC

CUDA error, related to multiprocessing

Open
#4 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
0
PR merge metrics
No merged PRs in 30d

Description

I'm tried to run LightMHC with Cuda 11.8.0

Command:

```bash
python LightMHC/lightmhc/inference.py data.input_csv_path=xray.lightmhc.csv data.output_dir=lighmhc-xray model.n_cpus=32 model.use_gpu=true model.batch_size=64
```

I got an error message:

```python
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/baakmanc/LightMHC/lightmhc/inference.py", line 77, in workflow
model = model.to(device)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 673, in to
return self._apply(convert)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 387, in _apply
module._apply(fn)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 387, in _apply
module._apply(fn)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 387, in _apply
module._apply(fn)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 409, in _apply
param_applied = fn(param)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 671, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/cuda/__init__.py", line 160, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
"""
```

I found that the start method can be set to 'spawn' by following the instructions here:
https://pytorch.org/docs/stable/notes/multiprocessing.html

That fixed the issue for me. Just letting you know.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.