LengthGroupedSampler会导致loss周期性振荡
Open
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 448
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 26
Description
SFT [InternVL2](https://github.com/InternLM/xtuner/pull/737) 时发现使用LengthGroupedSampler loss曲线会周期性振荡，使用DefaultSampler不会有这个问题。请问这是什么原因呢？
[类似的问题](https://ai.stackexchange.com/questions/41792/what-causes-my-loss-curve-to-consistently-oscillate-when-training-an-llm)
Contributor guide
Assessment
This issue has not been assessed yet.