PPI distributed training 收敛问题
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
单worker收敛情况:
INFO:tensorflow:f1 = 0.5928368, loss = 0.43671727, step = 1920 (2.334 sec)
INFO:tensorflow:f1 = 0.5933435, loss = 0.45692495, step = 1940 (2.293 sec)
INFO:tensorflow:f1 = 0.593776, loss = 0.41395587, step = 1960 (2.331 sec)
INFO:tensorflow:f1 = 0.59379023, loss = 0.43308347, step = 1980 (2.453 sec)
INFO:tensorflow:f1 = 0.59410304, loss = 0.43284538, step = 2000 (2.342 sec)
INFO:tensorflow:**f1 = 0.59446037,** loss = 0.41725546, step = 2020 (2.369 sec)
2 workers收敛就出现了问题,在迭代相同情况下,f1比单worker差很多。
INFO:tensorflow:f1 = 0.52126294, loss = 0.4985747, step = 1855 (2.872 sec)
INFO:tensorflow:f1 = 0.5215076, loss = 0.50285107, step = 1895 (2.710 sec)
INFO:tensorflow:f1 = 0.5229915, loss = 0.4858875, step = 1935 (2.710 sec)
INFO:tensorflow:f1 = 0.5244535, loss = 0.665834, step = 1977 (2.449 sec)
INFO:tensorflow:**f1 = 0.5231028**, loss = 0.57733107, step = 2017 (2.375 sec)
command:
python -m tf_euler --job_name=worker --task_index=0 --ps_hosts localhost:1999 --worker_hosts=localhost:2000 --model_dir ~/ppi1-512 --data_dir hdfs://localhost:4444/ppi4/ --max_id 56944 --feature_idx 1 --feature_dim 50 --label_idx 0 --label_dim 121 --model graphsage_supervised --mode train --num_epochs 30
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported command with one worker and two workers, comparing the logged F1 and loss at similar steps. Read the distributed training entry points used by the tf_euler module and the graphsage_supervised model; done means the cause is identified and two-worker training converges comparably or the remaining limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100