alibaba / alibaba/euler

PPI distributed training 收敛问题

Open
#198 18 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

单worker收敛情况:
INFO:tensorflow:f1 = 0.5928368, loss = 0.43671727, step = 1920 (2.334 sec)
INFO:tensorflow:f1 = 0.5933435, loss = 0.45692495, step = 1940 (2.293 sec)
INFO:tensorflow:f1 = 0.593776, loss = 0.41395587, step = 1960 (2.331 sec)
INFO:tensorflow:f1 = 0.59379023, loss = 0.43308347, step = 1980 (2.453 sec)
INFO:tensorflow:f1 = 0.59410304, loss = 0.43284538, step = 2000 (2.342 sec)
INFO:tensorflow:**f1 = 0.59446037,** loss = 0.41725546, step = 2020 (2.369 sec)

2 workers收敛就出现了问题,在迭代相同情况下,f1比单worker差很多。
INFO:tensorflow:f1 = 0.52126294, loss = 0.4985747, step = 1855 (2.872 sec)
INFO:tensorflow:f1 = 0.5215076, loss = 0.50285107, step = 1895 (2.710 sec)
INFO:tensorflow:f1 = 0.5229915, loss = 0.4858875, step = 1935 (2.710 sec)
INFO:tensorflow:f1 = 0.5244535, loss = 0.665834, step = 1977 (2.449 sec)
INFO:tensorflow:**f1 = 0.5231028**, loss = 0.57733107, step = 2017 (2.375 sec)

command:
python -m tf_euler --job_name=worker --task_index=0 --ps_hosts localhost:1999 --worker_hosts=localhost:2000 --model_dir ~/ppi1-512 --data_dir hdfs://localhost:4444/ppi4/ --max_id 56944 --feature_idx 1 --feature_dim 50 --label_idx 0 --label_dim 121 --model graphsage_supervised --mode train --num_epochs 30

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported command with one worker and two workers, comparing the logged F1 and loss at similar steps. Read the distributed training entry points used by the tf_euler module and the graphsage_supervised model; done means the cause is identified and two-worker training converges comparably or the remaining limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.