deepspeedai / deepspeedai/DeepSpeed
DeepSpeed FP16 causes bert pretrain ACC decline or NAN?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
We use DeepSpeed (v0.4.0) FP16 to train Bert Large, we found some Acc decline cases or NAN cases.
Like this picture:
And this picture:
When we set FP16 config to false or use AMP. The Acc convergence is good.
DeepSpeed FP16 implementation is like apex O2, there is some convergence issue about amp O2.
Is there some solution to the convergence problem of DeepSpeed FP16?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported BERT Large training case with DeepSpeed v0.4.0 and the shown FP16 configuration, then compare it with FP16 disabled or AMP as described. The issue mentions no files or tests; done would require a reproducible convergence or NaN case, a narrowed cause, and a validated fix or documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100