分布式训练的时候出现问题
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
这是worker的log,好像是worker和ps连不上?
```
Colocations handled automatically by placer.
58 WARNING:tensorflow:From /home/byli/.conda/envs/by/lib/python2.7/site-packages/tensorflow/python/ops/metrics_impl.p y:1472: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
59 Instructions for updating:
60 Use tf.cast instead.
61 WARNING:tensorflow:From tf_euler/python/metrics.py:31: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
62 Instructions for updating:
63 Deprecated in favor of operator or tf.math.divide.
64 WARNING:tensorflow:From /home/byli/.conda/envs/by/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py:30 66: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
65 Instructions for updating:
66 Use tf.cast instead.
67 INFO:tensorflow:Create CheckpointSaverHook.
68 SLF4J: Class path contains multiple SLF4J bindings.
69 SLF4J: Found binding in [jar:file:/data/home/byli/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/o rg/slf4j/impl/StaticLoggerBinder.class]
70 SLF4J: Found binding in [jar:file:/data/home/tati/apps/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.j ar!/org/slf4j/impl/StaticLoggerBinder.class]
71 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
72 SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
73 INFO:tensorflow:Graph was finalized.
74 WARNING: An illegal reflective access operation has occurred
75 WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/data/home /byli/hadoop-2.8.5/share/hadoop/common/lib/hadoop-auth-2.8.5.jar) to method sun.security.krb5.Config.getInstance() 76 WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.Kerbe rosUtil
77 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
78 WARNING: All illegal access operations will be denied in a future release
79 19/08/15 12:46:11 WARN hdfs.DFSClient: zero
80 WARNING: Logging before InitGoogleLogging() is written to STDERR
81 I0815 12:46:12.170971 27747 graph_builder.cc:81] Load Done: hdfs://localhost:9000/by/ppi/ppi_data.dat
82 I0815 12:46:12.172276 27618 graph_builder.cc:127] Each Thread Load Finish! Node Count:56944 Edge Count:1612348
83 I0815 12:46:12.172344 27618 graph_builder.cc:135] Graph Loading Finish!
84 I0815 12:46:12.584734 27618 graph_builder.cc:147] Graph Load Finish! Node Count:56944 Edge Count:1612348
85 I0815 12:46:12.597359 27618 graph_builder.cc:152] Done: build node sampler
86 I0815 12:46:12.597375 27618 graph_builder.cc:162] Graph build finish
87 I0815 12:46:12.597398 27618 graph_service.cc:179] service init finish
88 I0815 12:46:12.597888 27618 graph_service.cc:131] bound port: 127.0.1.1:43101
89 W0815 12:46:12.602923 27618 graph.h:198] global sampler is not ok
90 I0815 12:46:12.603827 27618 graph_service.cc:146] service start
91 I0815 12:46:12.604326 27638 zk_server_monitor.cc:238] Online node: 0#127.0.1.1:43101.
92 2019-08-15 12:46:20.094357: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
93 I0815 12:46:30.003481 27638 zk_server_monitor.cc:253] Offline node: 0#127.0.1.1:46829.
94 2019-08-15 12:46:30.094647: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
95 2019-08-15 12:46:40.094918: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
96 2019-08-15 12:46:50.095207: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
97 2019-08-15 12:47:00.095481: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
98 2019-08-15 12:47:10.095776: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
99 2019-08-15 12:47:20.096048: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
100 2019-08-15 12:47:30.096347: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
101 2019-08-15 12:47:40.096589: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
102 2019-08-15 12:47:50.096816: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
103 2019-08-15 12:48:00.097025: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
104 2019-08-15 12:48:10.097298: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
105 2019-08-15 12:48:20.097546: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
106 2019-08-15 12:48:30.097825: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
107 2019-08-15 12:48:40.098092: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
108 2019-08-15 12:48:50.098370: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
109 2019-08-15 12:49:00.098632: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
110 2019-08-15 12:49:10.098926: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
111 2019-08-15 12:49:20.099202: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
112 2019-08-15 12:49:30.099501: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
113 2019-08-15 12:49:40.099770: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
114 2019-08-15 12:49:50.100028: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
115 2019-08-15 12:50:00.100301: I tensorflow/core/distributed_runtime/master.cc:267] CreateSession still waiting for r esponse from worker: /job:ps/replica:0/task:0
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the worker log around graph_service.cc, zk_server_monitor.cc, and TensorFlow's CreateSession message, then trace how the worker and ps endpoints are configured. Reproduce the distributed-training run and determine why the ps worker remains unreachable; done means the session is created without repeated waiting messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, hadoop, python, tensorflow
- Domain
- distributed-systems, machine-learning, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100