alibaba / alibaba/euler

Euler2.0分布式训练遇到问题

Open
#294 18 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

我先贴上我的分布式运行脚本:
我先跑了example/gat/run_gat.py 成功,过程中产生了 cora文件,里面 包含euler/Node euler/Edge等,各包含10个.dat 文件
我将euler文件作为data_dir
```
def main(_):
flags_obj = tf.flags.FLAGS
if flags_obj.local:
local_test()
if flags_obj.tfdist:
job_name = flags_obj.job_name
if job_name == "start_euler":
start_euler()
elif job_name == "worker":
start_tf()
elif job_name == "ps":
start_tf()

def start_euler():
flags_obj = tf.flags.FLAGS
# ------------- euler start ----------------
data_dir = flags_obj.data_dir # cora/euler文件
shard_idx = flags_obj.shard_idx
shard_num = flags_obj.shard_num
zk_addr = flags_obj.zk_addr
zk_path = flags_obj.zk_path
euler.start(
directory=data_dir, # 图数据路径 生成二进制的euler文件
shard_idx=shard_idx, # 当前启动的进程为k号shard
shard_num=shard_num, # 一共有N个shard
zk_addr=zk_addr, # Zookeeper address, ip:port
zk_path=zk_path, # Zookeeper path
#module=euler.Module.DEFAULT_MODULE)
module=(euler.Module.NODE | euler.Module.EDGE))
while True:
time.sleep(1)

def start_tf():
...
tf_config = {
'cluster': {'chief': chief_hosts, 'worker': worker_hosts, 'ps': ps_hosts},
'task': {'type': job_name, 'index': task_index}
}
os.environ['TF_CONFIG'] = json.dumps(tf_config)
model = ... # 省略
params = {...}
if not job_name == "ps":
print("init graph........")
tf_euler.initialize_graph({
'mode': 'remote',
'zk_server': zk_addr,
'zk_path': zk_path,
'shard_num': shard_num,
'num_retries': 1
})

# ------------init estimator---------------
print("start train...")
tf.logging.set_verbosity(tf.logging.INFO)
# Estimator的创建与训练/验证/预测
config = tf.estimator.RunConfig(log_step_count_steps=None)
base_estimator = NodeEstimator(model, params, config)
base_estimator.train_and_evaluate()

```
然后 在本地模拟tf的分布式
```
# dist 两个机器负责load数据, 一个chief 一个worker 一个ps
python gat_euler_htj.py --tfdist=True --job_name='start_euler' --shard_idx=0 --shard_name=2 --data_dir="/workspace/tony_test/htj_gat/cora_dist/euler"
python gat_euler_htj.py --tfdist=True --job_name='start_euler' --shard_idx=1 --shard_name=2 --data_dir="/workspace/tony_test/htj_gat/cora_dist/euler"
# tf 分布式 cluster
python gat_euler_htj.py --tfdist=True --job_name='worker' --task_index=0 --ps_hosts = "localhost:2222" --worker_hosts="localhost:2223, localhost:2224" # chief
python gat_euler_htj.py --tfdist=True --job_name='worker' --task_index=1 --ps_hosts = "localhost:2222" --worker_hosts="localhost:2223, localhost:2224" # worker
python gat_euler_htj.py --tfdist=True --job_name='ps' --task_index=0 --ps_hosts = "localhost:2222" --worker_hosts="localhost:2223, localhost:2224" # ps
```
然后挨个执行
其中: load数据的两个机器 成功: server started successfully
ps 成功: Started server with target: grpc://localhost:2222
worker 和chief的机器失败:
报错信息: ```E0802 12:06:26.013273386 55381 completion_queue.cc:469] assertion failed: cq_event_queue_num_items(&cqd->queue) == 0 已放弃 (core dumped)```

想问一下 这是什么原因,是这个脚本代码有问题吗?还是euler的问题?能够提供解决思路呢?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with example/gat/run_gat.py and the distributed commands for gat_euler_htj.py, then trace tf_euler.initialize_graph and the Estimator startup for the worker and chief. Reproduce the grpc completion_queue.cc assertion while comparing the successful Euler and ps processes. Done means identifying whether the failure is in the script or Euler and documenting a reproducible cause and resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python, tensorflow
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.