Cannot use multiple workers when training KPConv.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 365
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 1
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).
### Describe the issue
Training the KPConv model cannot use multiple workers.
The following error was found.
Original Traceback (most recent call last):
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 552, in collate_fn
batching_result.to(self.device)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in to
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/cuda/__init__.py", line 160, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
From the log, I found that the concated batches were put into GPU directly in collate_fn. By contrast, for other models, the batches on CPU were returned.

May I know why only this model had this kind of setting? Currently, I can only set num_workers to 0 to train the KPConv successfully. But the efficiency is not that high due to 0 worker.
And I have to set pin_memory to false, since all the sampled data has been already put into GPU when concating the batches.
When I removed the "batching_result.to(self.device)", another error occured.
RecursionError: maximum recursion depth exceeded
RuntimeError: Pin memory thread exited unexpectedly
In summary, same settings (num_workers=8 and pin_memory = true) can work for other models like RandLA and point Transformer but not for KPConv.
### Steps to reproduce the bug
```python
from ml3d.torch.models import kpconv
from ml3d.torch.models import randlanet
from ml3d.torch.pipelines import semantic_segmentation
from ml3d.datasets.s3dis import *
import torch
import open3d.ml as _ml3d
import sys, os
if __name__ == '__main__':
os.chdir(sys.path[0])
cfg_file = "./ml3d/configs/kpconv_s3dis.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)
dataset = S3DIS(**cfg.dataset)
model=kpconv.KPFCNN(**cfg.model)
pipeline = semantic_segmentation.SemanticSegmentation(model=model, dataset=dataset,**cfg.pipeline)
pipeline.run_train()
```
### Error message
Traceback (most recent call last):
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/KPConvTraining.py", line 35, in
pipeline.run_train()
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/pipelines/semantic_segmentation.py", line 408, in run_train
for step, inputs in enumerate(tqdm(train_loader, desc='training')):
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
for obj in iterable:
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
data = self._next_data()
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
return self._process_data(data)
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
data.reraise()
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 552, in collate_fn
batching_result.to(self.device)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in to
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/cuda/__init__.py", line 160, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
### Expected behavior
Multiple workers can be used for training KPConv.
### Open3D, Python and System information
```markdown
- Operating system: (WSL Ubuntu 18.04 on Windows 10 64-bit)
- Python version: (Python 3.8)
- Open3D version: (0.15.2)
- Is this remote workstation?: yes
- How did you install Open3D?: (pip)
```
### Additional information
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.