deepspeedai / deepspeedai/DeepSpeed
max_grad_norm is ignored in FP16 training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I am currently fighting with a dynamic loss scale that is constantly decreasing due to gradient overflows. Setting max_grad_norm in the JSON config has no effect since it is overidden in deepspeed_light.py https://github.com/microsoft/DeepSpeed/blob/13fd3dca2abb6d6a6af62d46457fff8a1a678a4a/deepspeed/pt/deepspeed_light.py#L407-L408:
I think this modification should be removed.
Kind regards
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in deepspeed/pt/deepspeed_light.py around lines 407-408 and trace how max_grad_norm from the JSON config is handled during FP16 training. Remove the override described in the issue, then verify that the configured value is honored and no longer causes the reported loss-scale behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100