deepspeedai / deepspeedai/DeepSpeed
[BUG] 1-bit LAMB not compatible with bf16
Open
Nobody has claimed this yet.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
When training with 1-bit LAMB, the DeepSpeed ModelEngine complains that loss scaling is not enabled. However bf16 does not require loss scaling so does not support this (hence it is not enabled). The code should be updated to allow an exception for bf16.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the DeepSpeed ModelEngine path used by 1-bit LAMB and reproduce the failure with bf16 training. Trace where loss scaling is required and compare that check with bf16 configuration, then verify that bf16 1-bit LAMB runs without loss scaling while other configurations retain their existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100