dmlc / dmlc/dgl

Distributed multi GPU training example missing ?

Open
#4,502 4 comments 0 reactions 2 assignees Claimed by @peizhou001 View on GitHub
feature request has workaround topic: Distributed DGL topic: doc
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 📚 Documentation

Hi, thanks a lot for the great work.

I dug quite a bit every where in the doc but could not manage to find how to do what I tried to do.

I am trying to do distributed multi GPU training, ie use two computer with let say 2 GPUs each, and I think this is not covered in the documentation.

First way:
- starting from distributed training example https://docs.dgl.ai/tutorials/dist/1_node_classification.html with 2 parts
- then spawning on every machine several processes by hand (using example from https://docs.dgl.ai/tutorials/multi/2_node_classification.html)
- this does not work because every spawned process on a given machine tries to bind to same port to communicate to the server (if I understood correctly the error message)

Second way :
- starting from distributed example (with 4 parts), simply try to give same ip with different ports ie in ip_config.txt something like
192.168.0.1 30050
192.168.0.1 30051
192.168.0.2 30050
192.168.0.2 30051
- but this does not wok either, it seems that 30050 is not used at all, as if port value was unique for each IP (ie a map)

Third way :
- give launch.py a value of 2 for num_trainers options, and hope that it magically does the stuff
- processes are swpaned and visible on every GPUs, but
- I get some obscure nccl init error and nothing more happens

I wonder if there could be some documentation on how to do this, I am starting to be totally clueless.
Thanks again for the amazing work done here

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.