deepspeedai / deepspeedai/DeepSpeed
[BUG]Loss decreases cyclically with epoch as the cycle
Open
Nobody has claimed this yet.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I use deepspeed to train my model and start training by loading the original trained model. I found that my loss takes epoch as a cycle, cyclically, falling first and then rising, and overall it is declining.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, entry points, training configuration, or reproducible example are mentioned. Start by collecting the DeepSpeed configuration, model and optimizer details, loss logs, and environment information, then reproduce the cyclic loss across epochs. Done means identifying a reproducible cause and adding a regression test or clearly documented fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100