dmlc / dmlc/dgl

Have a proper way of changing DGL_DIST_MAX_TRY_TIMES in the launch script.

Open
#4,580 4 comments 0 reactions 1 assignee Claimed by @peizhou001 View on GitHub
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🔨Work Item

**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.

Project tracker: https://github.com/orgs/dmlc/projects/2

## Description

There are two issues here:
* The launch script doesn't understand DGL_DIST_MAX_TRY_TIMES. When running distributed training on a large graph, we need to change DGL_DIST_MAX_TRY_TIMES. We need a user-friendly way to change the environment variable and document it.
* Even if I change the environment variable as follows
```
python3 -u ${LAUNCH_PATH} \
--workspace ${WORKSPACE} \
--num_trainers ${NUM_TRAINERS} \
--num_servers ${NUM_SERVERS} \
--num_samplers ${NUM_SAMPLERS} \
--part_config ${PART_CONFIG} \
--ip_config ${IP_CONFIG} \
--ssh_port 2222 \
--graph_format csc,coo \
"DGL_DIST_MAX_TRY_TIMES=10240 python3 m5gnn_lp_huggingface.py --cf ${TRAINING_CONFIG} --feat-name feat --train-nodes 0"
```
I still get the following errors:
```
Traceback (most recent call last):
File "m5gnn_lp_huggingface.py", line 26, in
main(args)
File "m5gnn_lp_huggingface.py", line 14, in main
trainer = M5gnnLinkPredictionTrainer(config, m5_models)
File "/m5-gnn/python/m5gnn/trainer/lp_trainer.py", line 66, in __init__
self.init_dist_context(config.ip_config,
File "/m5-gnn/python/m5gnn/trainer/m5gnn_trainer.py", line 22, in init_dist_context
dgl.distributed.initialize(ip_config, net_type='socket')
File "/usr/local/lib/python3.8/dist-packages/dgl/distributed/dist_context.py", line 267, in initialize
connect_to_server(ip_config, num_servers, max_queue_size, net_type, group_id=group_id)
File "/usr/local/lib/python3.8/dist-packages/dgl/distributed/rpc_client.py", line 182, in connect_to_server
raise rpc.DistConnectError(max_try_times)
dgl.distributed.rpc.DistConnectError: Failed to build conncetion with peer after 1024 retries. Please check network availability or increase max try times via 'DGL_DIST_MAX_TRY_TIMES'.
```

## Depending work items or issues

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.