分布式训练为什么出现 Variables not initialized?
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
hello,咨询一下,我们进行分布式训练时(1 PS + 2 Worker),跑的是官网ppi数据模型,启动worker-0启动后会立刻才是计算,如下:
==> /tmp/log.worker.0 <==
INFO:tensorflow:f1 = 0.40184546, loss = 0.69358194, step = 0
INFO:tensorflow:f1 = 0.4026365, loss = 0.5608437, step = 20 (1.201 sec)
INFO:tensorflow:f1 = 0.4041432, loss = 0.53496814, step = 40 (1.069 sec)
INFO:tensorflow:f1 = 0.4073638, loss = 0.5300987, step = 60 (1.115 sec)
但是,worker-1启动后,会有一段等待时间,然后才开始进行计算,具体日志如下:
INFO:tensorflow:Graph was finalized.
2019-04-23 03:45:56.103402: I tensorflow/core/distributed_runtime/master_session.cc:1192] Start master session 27c7c5a9b0b92ad1 with config: gpu_options { allow_growth: true }
INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: Variables not initialized: supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_3/kernel, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_5/kernel, supervisedgraphsage_1/dense_1/kernel, supervisedgraphsage_1/dense_1/bias, global_step, beta1_power, beta2_power, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_2/kernel/Adam_1, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_3/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_1/dense_3/kernel/Adam_1, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_4/kernel/Adam_1, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_5/kernel/Adam, supervisedgraphsage_1/sageencoder_1/meanaggregator_2/dense_5/kernel/Adam_1, supervisedgraphsage_1/dense_1/kernel/Adam, supervisedgraphsage_1/dense_1/kernel/Adam_1, supervisedgraphsage_1/dense_1/bias/Adam, supervisedgraphsage_1/dense_1/bias/Adam_1, num_finished_workers, ready: None
从日志看到worker-1貌似在等待模型变量初始化:
INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: Variables not initialized
请教一下,这个问题怎么解决呢?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the official PPI model with one parameter server and two workers, then compare the worker-0 and worker-1 startup logs. Investigate the distributed initialization path responsible for “Waiting for model to be ready” and the listed uninitialized variables; done means the cause of the delay is identified and a verified resolution or documented limitation is provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100