Euler2.0分布式训练遇到问题
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
我先贴上我的分布式运行脚本:
我先跑了example/gat/run_gat.py 成功,过程中产生了 cora文件,里面 包含euler/Node euler/Edge等,各包含10个.dat 文件
我将euler文件作为data_dir
```
def main(_):
flags_obj = tf.flags.FLAGS
if flags_obj.local:
local_test()
if flags_obj.tfdist:
job_name = flags_obj.job_name
if job_name == "start_euler":
start_euler()
elif job_name == "worker":
start_tf()
elif job_name == "ps":
start_tf()
def start_euler():
flags_obj = tf.flags.FLAGS
# ------------- euler start ----------------
data_dir = flags_obj.data_dir # cora/euler文件
shard_idx = flags_obj.shard_idx
shard_num = flags_obj.shard_num
zk_addr = flags_obj.zk_addr
zk_path = flags_obj.zk_path
euler.start(
directory=data_dir, # 图数据路径 生成二进制的euler文件
shard_idx=shard_idx, # 当前启动的进程为k号shard
shard_num=shard_num, # 一共有N个shard
zk_addr=zk_addr, # Zookeeper address, ip:port
zk_path=zk_path, # Zookeeper path
#module=euler.Module.DEFAULT_MODULE)
module=(euler.Module.NODE | euler.Module.EDGE))
while True:
time.sleep(1)
def start_tf():
...
tf_config = {
'cluster': {'chief': chief_hosts, 'worker': worker_hosts, 'ps': ps_hosts},
'task': {'type': job_name, 'index': task_index}
}
os.environ['TF_CONFIG'] = json.dumps(tf_config)
model = ... # 省略
params = {...}
if not job_name == "ps":
print("init graph........")
tf_euler.initialize_graph({
'mode': 'remote',
'zk_server': zk_addr,
'zk_path': zk_path,
'shard_num': shard_num,
'num_retries': 1
})
# ------------init estimator---------------
print("start train...")
tf.logging.set_verbosity(tf.logging.INFO)
# Estimator的创建与训练/验证/预测
config = tf.estimator.RunConfig(log_step_count_steps=None)
base_estimator = NodeEstimator(model, params, config)
base_estimator.train_and_evaluate()
```
然后 在本地模拟tf的分布式
```
# dist 两个机器负责load数据, 一个chief 一个worker 一个ps
python gat_euler_htj.py --tfdist=True --job_name='start_euler' --shard_idx=0 --shard_name=2 --data_dir="/workspace/tony_test/htj_gat/cora_dist/euler"
python gat_euler_htj.py --tfdist=True --job_name='start_euler' --shard_idx=1 --shard_name=2 --data_dir="/workspace/tony_test/htj_gat/cora_dist/euler"
# tf 分布式 cluster
python gat_euler_htj.py --tfdist=True --job_name='worker' --task_index=0 --ps_hosts = "localhost:2222" --worker_hosts="localhost:2223, localhost:2224" # chief
python gat_euler_htj.py --tfdist=True --job_name='worker' --task_index=1 --ps_hosts = "localhost:2222" --worker_hosts="localhost:2223, localhost:2224" # worker
python gat_euler_htj.py --tfdist=True --job_name='ps' --task_index=0 --ps_hosts = "localhost:2222" --worker_hosts="localhost:2223, localhost:2224" # ps
```
然后挨个执行
其中: load数据的两个机器 成功: server started successfully
ps 成功: Started server with target: grpc://localhost:2222
worker 和chief的机器失败:
报错信息: ```E0802 12:06:26.013273386 55381 completion_queue.cc:469] assertion failed: cq_event_queue_num_items(&cqd->queue) == 0 已放弃 (core dumped)```
想问一下 这是什么原因,是这个脚本代码有问题吗?还是euler的问题?能够提供解决思路呢?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with example/gat/run_gat.py and the distributed commands for gat_euler_htj.py, then trace tf_euler.initialize_graph and the Estimator startup for the worker and chief. Reproduce the grpc completion_queue.cc assertion while comparing the successful Euler and ps processes. Done means identifying whether the failure is in the script or Euler and documenting a reproducible cause and resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python, tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100