Have a proper way of changing DGL_DIST_MAX_TRY_TIMES in the launch script.
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🔨Work Item
**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.
Project tracker: https://github.com/orgs/dmlc/projects/2
## Description
There are two issues here:
* The launch script doesn't understand DGL_DIST_MAX_TRY_TIMES. When running distributed training on a large graph, we need to change DGL_DIST_MAX_TRY_TIMES. We need a user-friendly way to change the environment variable and document it.
* Even if I change the environment variable as follows
```
python3 -u ${LAUNCH_PATH} \
--workspace ${WORKSPACE} \
--num_trainers ${NUM_TRAINERS} \
--num_servers ${NUM_SERVERS} \
--num_samplers ${NUM_SAMPLERS} \
--part_config ${PART_CONFIG} \
--ip_config ${IP_CONFIG} \
--ssh_port 2222 \
--graph_format csc,coo \
"DGL_DIST_MAX_TRY_TIMES=10240 python3 m5gnn_lp_huggingface.py --cf ${TRAINING_CONFIG} --feat-name feat --train-nodes 0"
```
I still get the following errors:
```
Traceback (most recent call last):
File "m5gnn_lp_huggingface.py", line 26, in
main(args)
File "m5gnn_lp_huggingface.py", line 14, in main
trainer = M5gnnLinkPredictionTrainer(config, m5_models)
File "/m5-gnn/python/m5gnn/trainer/lp_trainer.py", line 66, in __init__
self.init_dist_context(config.ip_config,
File "/m5-gnn/python/m5gnn/trainer/m5gnn_trainer.py", line 22, in init_dist_context
dgl.distributed.initialize(ip_config, net_type='socket')
File "/usr/local/lib/python3.8/dist-packages/dgl/distributed/dist_context.py", line 267, in initialize
connect_to_server(ip_config, num_servers, max_queue_size, net_type, group_id=group_id)
File "/usr/local/lib/python3.8/dist-packages/dgl/distributed/rpc_client.py", line 182, in connect_to_server
raise rpc.DistConnectError(max_try_times)
dgl.distributed.rpc.DistConnectError: Failed to build conncetion with peer after 1024 retries. Please check network availability or increase max try times via 'DGL_DIST_MAX_TRY_TIMES'.
```
## Depending work items or issues
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.