Slow training using more CPU rather than GPU in docker container
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 365
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 1
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).
### My Question
Hello!
Thank you for your codes. I have successfully implemented your framework to train several models in a Docker container.
Following the readme, I set the Docker environment as follows,
Python 3.8.8+Open3d 0.15.2+pytorch 1.8.2+cuda11.1+cudnn8
device="GPU" in semantic_segmentation.py.
However, training of a KPConv model on the S3DIS dataset encountered errors.
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
I set "torch.multiprocessing.set_start_method('spawn')" but another error was given.
RuntimeError: cuda runtime error (801) : operation not supported at /pytorch/torch/csrc/generic/StorageSharing.cpp:253
When I changed the number of workers to 0, there would exist no errors. Thus, I guess, it is the shared memory issue. But actually, I had set "--ipc=host" for the docker environment.
Another finding was that training of the RandLA Net model on the S3DIS dataset could be successfully implemented with n>0 workers, but the training efficiency was still slow. CPU was highly used but nearly no GPU utilization.
On the contrary, when I used this same context to train KPConv codes provided by the paper author, the utilization of GPU was stable and high. So it seems the program of the open3dml framework could not use the GPU sources well in docker.
Can anyone give me some suggestions on how to improve it?
Cheers,
Eric.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.