alibaba / alibaba/FastNN

2台服务器分布式跑resnet_split.py遇到无限等待的情况

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
85
Forks
20
PR merge metrics
No merged PRs in 30d

Description

**环境:** nvcr.io/nvidia/tensorflow:21.12-tf1-py3镜像的容器
**代码:** FastNN/resnet/resnet_split.py
**执行命令:**
服务器1:TF_CONFIG='{"cluster":{"worker":["172.20.21.181:55375","172.20.21.189:55376"]},"task":{"type":"worker","index":0}}' bash scripts/train_split.sh
服务器2:TF_CONFIG='{"cluster":{"worker":["172.20.21.181:55375","172.20.21.189:55376"]},"task":{"type":"worker","index":1}}' bash scripts/train_split.sh

服务器1的执行情况:
![image](https://github.com/alibaba/FastNN/assets/55943192/58d97a8f-fa61-4239-a70b-fd8d1c4ba58b)
服务器2的执行情况:
![image](https://github.com/alibaba/FastNN/assets/55943192/f8b7b791-98f6-4fb3-ac4f-3f979f64ee7f)

可以看到服务器1的still waiting只打印了2条就不打印了说明已经接收到了服务器2的回复,但是没有继续往下运行。
**补充:** 同样的环境可以分布式运行bert,服务器之间是可以正常连接跑分布式训练的。

想问下是我的执行问题还是代码需要进行修改?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with FastNN/resnet/resnet_split.py and scripts/train_split.sh, then reproduce the two-server run using the TF_CONFIG values and commands in the report. Compare the waiting behavior with the reported distributed BERT run and inspect the available output for where progress stops. Done means identifying whether the launch configuration or code causes the hang and validating the result with a successful two-server run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.