请问多ps 多euler的情况下有infer的例子可以借鉴吗
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
目前处于分布式训练保存ckpt,可以读取 可以再训练的状态, 当user_id数量极多的情况下模型参数非常大 应该是多个ps来扛,这个时候想要导出user_embedding 不知有没有例子可以参考 ,我尝试在examples/graphsage/run_graphsage.py的基础上进行了 分布式train的修改是ok的, 不过在此基础上直接调用model_estimator.infer() 好像并不行,wo
进行infer时 代码、执行语句、日志分别是
**代码主要部分**
`tf_config={
'cluster': {'chief': chief_hosts, 'worker': worker_hosts, 'ps': ps_hosts},
'task': {'type': job_name, 'index': task_index}
}
if job_name == 'worker' and task_index == 0:
tf_config['task'] = {"index": 0, "type": "chief"}
....
....
model = graphSage的例子
config = tf.estimator.RunConfig(log_step_count_steps=None)
model_estimator = NodeEstimator(model, params, config)
if flags_obj.run_mode == 'train':
model_estimator.train_and_evaluate()
elif flags_obj.run_mode == 'evaluate':
model_estimator.evaluate()
elif flags_obj.run_mode == 'infer':
model_estimator.infer()
else:
raise ValueError('Run mode not exist!')
`
**执行语句**
python run_graphsage_distribute_new.py --job_name 'start_euler' --shard_idx 0 --shard_num ${shard_num} --data_dir ${data_dir} --zk_addr ${zk_addr} --zk_path ${zk_path}
python run_graphsage_distribute_new.py --job_name 'start_euler' --shard_idx 1 --shard_num ${shard_num} --data_dir ${data_dir} --zk_addr ${zk_addr} --zk_path ${zk_path}
python run_graphsage_distribute_new.py --job_name 'ps' --shard_num ${shard_num} --task_index 0 --ps_hosts ${ps_hosts} --worker_hosts ${worker_hosts} --chief_hosts ${chief_hosts} --zk_addr ${zk_addr} --zk_path ${zk_path} # ps
刚启动到这个ps 时就已经有问题了(目前仅1个ps节点)
**日志图片**

Contributor guide
No contributing guide indexed for this repository
Research direction
Start with examples/graphsage/run_graphsage.py, the NodeEstimator.infer() entry point, and the supplied distributed launch commands. Reproduce the multi-PS startup and inference behavior, then inspect the logs to determine the failure boundary. Done means distributed inference can start and user embeddings can be exported from the checkpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100