isl-org / isl-org/Open3D-ML

Cannot use multiple workers when training KPConv.

Open
#524 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.3k
Forks
365
Avg merge
4h 6m
Merged PRs (30d)
1

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

Training the KPConv model cannot use multiple workers.

The following error was found.

Original Traceback (most recent call last):
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 552, in collate_fn
batching_result.to(self.device)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in to
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/cuda/__init__.py", line 160, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

From the log, I found that the concated batches were put into GPU directly in collate_fn. By contrast, for other models, the batches on CPU were returned.

![image](https://user-images.githubusercontent.com/35157984/164980709-cef33e68-a41d-4fa4-a3a1-e28975a7ff5a.png)

May I know why only this model had this kind of setting? Currently, I can only set num_workers to 0 to train the KPConv successfully. But the efficiency is not that high due to 0 worker.

And I have to set pin_memory to false, since all the sampled data has been already put into GPU when concating the batches.

When I removed the "batching_result.to(self.device)", another error occured.

RecursionError: maximum recursion depth exceeded
RuntimeError: Pin memory thread exited unexpectedly

In summary, same settings (num_workers=8 and pin_memory = true) can work for other models like RandLA and point Transformer but not for KPConv.

### Steps to reproduce the bug

```python
from ml3d.torch.models import kpconv

from ml3d.torch.models import randlanet

from ml3d.torch.pipelines import semantic_segmentation

from ml3d.datasets.s3dis import *

import torch

import open3d.ml as _ml3d

import sys, os

if __name__ == '__main__':

os.chdir(sys.path[0])
cfg_file = "./ml3d/configs/kpconv_s3dis.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)
dataset = S3DIS(**cfg.dataset)

model=kpconv.KPFCNN(**cfg.model)

pipeline = semantic_segmentation.SemanticSegmentation(model=model, dataset=dataset,**cfg.pipeline)

pipeline.run_train()
```

### Error message

Traceback (most recent call last):
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/KPConvTraining.py", line 35, in
pipeline.run_train()
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/pipelines/semantic_segmentation.py", line 408, in run_train
for step, inputs in enumerate(tqdm(train_loader, desc='training')):
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
for obj in iterable:
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
data = self._next_data()
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
return self._process_data(data)
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
data.reraise()
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 552, in collate_fn
batching_result.to(self.device)
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in to
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/mnt/e/SoftwareHub/WSL/Ubuntu_18.04/Deeplearning/open3dmlNew/Open3D-ML-master/ml3d/torch/dataloaders/concat_batcher.py", line 328, in
self.points = [in_tensor.to(device) for in_tensor in self.points]
File "/root/anaconda3/envs/open3dml/lib/python3.8/site-packages/torch/cuda/__init__.py", line 160, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

### Expected behavior

Multiple workers can be used for training KPConv.

### Open3D, Python and System information

```markdown
- Operating system: (WSL Ubuntu 18.04 on Windows 10 64-bit)
- Python version: (Python 3.8)
- Open3D version: (0.15.2)
- Is this remote workstation?: yes
- How did you install Open3D?: (pip)
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.