在进行分布式训练的时候，loss在开始几步之后，变成一样的了，是什么原因呢？
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
INFO:tensorflow:loss = 110098360.0, mrr = 0.40143234, step = 267 (2.804 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 494 (2.631 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 712 (2.521 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 941 (2.626 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 1162 (2.563 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 1378 (2.481 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 1601 (2.502 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 1822 (2.501 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 2053 (2.539 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 2273 (2.489 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 2490 (2.448 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 2798 (2.735 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 3200 (2.809 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 3603 (2.766 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 4015 (2.896 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 4423 (2.844 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 4835 (2.836 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 5245 (2.868 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 5641 (2.788 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 6048 (2.842 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 6452 (2.787 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 6849 (2.749 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 7242 (2.718 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 7644 (2.756 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 8039 (2.743 sec)
INFO:tensorflow:loss = 2129.3452, mrr = 0.16666669, step = 8430 (2.666 sec)
Contributor guide
No contributing guide indexed for this repository
Research direction
The report only provides TensorFlow training logs and says the loss and MRR become constant during distributed training. Start by locating the distributed-training entry point and reproducing the logged behavior; inspect the training data and optimization flow. Done means identifying the cause and documenting or fixing the condition that makes the metrics remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100