alibaba / alibaba/euler

分布式训练为什么出现 Variables not initialized?

Open
#95 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

hello,咨询一下,我们进行分布式训练时(1 PS + 2 Worker),跑的是官网ppi数据模型,启动worker-0启动后会立刻才是计算,如下:
==> /tmp/log.worker.0 <==
INFO:tensorflow:f1 = 0.40184546, loss = 0.69358194, step = 0
INFO:tensorflow:f1 = 0.4026365, loss = 0.5608437, step = 20 (1.201 sec)
INFO:tensorflow:f1 = 0.4041432, loss = 0.53496814, step = 40 (1.069 sec)
INFO:tensorflow:f1 = 0.4073638, loss = 0.5300987, step = 60 (1.115 sec)

但是,worker-1启动后,会有一段等待时间,然后才开始进行计算,具体日志如下:
INFO:tensorflow:Graph was finalized.
2019-04-23 03:45:56.103402: I tensorflow/core/distributed_runtime/master_session.cc:1192] Start master session 27c7c5a9b0b92ad1 with config: gpu_options { allow_growth: true }
INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: Variables not initialized: supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_3/kernel, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_5/kernel, supervisedgraphsage_1/dense_1/kernel, supervisedgraphsage_1/dense_1/bias, global_step, beta1_power, beta2_power, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam_1, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_3/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_3/kernel/Adam_1, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam_1, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_5/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_5/kernel/Adam_1, supervisedgraphsage_1/dense_1/kernel/Adam, supervisedgraphsage_1/dense_1/kernel/Adam_1, supervisedgraphsage_1/dense_1/bias/Adam, supervisedgraphsage_1/dense_1/bias/Adam_1, num_finished_workers, ready: None

从日志看到worker-1貌似在等待模型变量初始化:
INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: Variables not initialized

请教一下,这个问题怎么解决呢?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the official PPI model with one parameter server and two workers, then compare the worker-0 and worker-1 startup logs. Investigate the distributed initialization path responsible for “Waiting for model to be ready” and the listed uninitialized variables; done means the cause of the delay is identified and a verified resolution or documented limitation is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
tensorflow
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.