isl-org / isl-org/Open3D-ML

Slow training using more CPU rather than GPU in docker container

Open
#504 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
2.3k
Forks
365
Avg merge
4h 6m
Merged PRs (30d)
1

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### My Question

Hello!
Thank you for your codes. I have successfully implemented your framework to train several models in a Docker container.

Following the readme, I set the Docker environment as follows,
Python 3.8.8+Open3d 0.15.2+pytorch 1.8.2+cuda11.1+cudnn8
device="GPU" in semantic_segmentation.py.

However, training of a KPConv model on the S3DIS dataset encountered errors.
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
I set "torch.multiprocessing.set_start_method('spawn')" but another error was given.
RuntimeError: cuda runtime error (801) : operation not supported at /pytorch/torch/csrc/generic/StorageSharing.cpp:253

When I changed the number of workers to 0, there would exist no errors. Thus, I guess, it is the shared memory issue. But actually, I had set "--ipc=host" for the docker environment.

Another finding was that training of the RandLA Net model on the S3DIS dataset could be successfully implemented with n>0 workers, but the training efficiency was still slow. CPU was highly used but nearly no GPU utilization.

On the contrary, when I used this same context to train KPConv codes provided by the paper author, the utilization of GPU was stable and high. So it seems the program of the open3dml framework could not use the GPU sources well in docker.

Can anyone give me some suggestions on how to improve it?

Cheers,

Eric.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.