Distributed multi GPU training example missing ?
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 📚 Documentation
Hi, thanks a lot for the great work.
I dug quite a bit every where in the doc but could not manage to find how to do what I tried to do.
I am trying to do distributed multi GPU training, ie use two computer with let say 2 GPUs each, and I think this is not covered in the documentation.
First way:
- starting from distributed training example https://docs.dgl.ai/tutorials/dist/1_node_classification.html with 2 parts
- then spawning on every machine several processes by hand (using example from https://docs.dgl.ai/tutorials/multi/2_node_classification.html)
- this does not work because every spawned process on a given machine tries to bind to same port to communicate to the server (if I understood correctly the error message)
Second way :
- starting from distributed example (with 4 parts), simply try to give same ip with different ports ie in ip_config.txt something like
192.168.0.1 30050
192.168.0.1 30051
192.168.0.2 30050
192.168.0.2 30051
- but this does not wok either, it seems that 30050 is not used at all, as if port value was unique for each IP (ie a map)
Third way :
- give launch.py a value of 2 for num_trainers options, and hope that it magically does the stuff
- processes are swpaned and visible on every GPUs, but
- I get some obscure nccl init error and nothing more happens
I wonder if there could be some documentation on how to do this, I am starting to be totally clueless.
Thanks again for the amazing work done here
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.