deepspeedai / deepspeedai/DeepSpeed
[BUG] bf16 w/ ZeRO Stage 0 assert
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
We need to be able to turn zero off, but it breaks:
assert not (self.bfloat16_enabled and (self.zero_optimization_stage not in {1, 2, 3})), f'bfloat16 mode is only enabled for Zero 1,2,3 currently. got {self.zero_optimization_stage}
So I think that assert should be removed altogether now, which I think will solve this issue.
Thanks.
@jeffra
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the configuration assertion shown in the issue and inspect how bfloat16 is handled when ZeRO is disabled at stage 0. Reproduce a bfloat16 configuration with ZeRO stage 0, then verify initialization proceeds without the assertion and that the existing ZeRO stages still behave correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100