deepspeedai / deepspeedai/DeepSpeed
static loss scale with stage 0 occured error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Config:
`
deepspeed_config = {
"fp16": {
"enabled": "auto",
"loss_scale": 4,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 0.00001,
"auto_cast": True
},
"zero_optimization": {
"stage": 0
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": "auto",
"betas": "auto",
"eps": "auto",
"weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
"warmup_min_lr": "auto",
"warmup_max_lr": "auto",
"warmup_num_steps": "auto"
}
},
"gradient_clipping": 'auto',
"train_batch_size": 'auto',
"train_micro_batch_size_per_gpu": 'auto'
}
`
Error be like :
File "/usr/local/miniconda3/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step self.optimizer.step() File "/usr/local/miniconda3/lib/python3.10/site-packages/deepspeed/runtime/fp16/fused_optimizer.py", line 255, in step self.timers(OVERFLOW_CHECK_TIMER).start() TypeError: 'NoneType' object is not callable
So may be we should initialize timer before line 1571?
Then I changed zero_optimization to stage 2, and the error didn't show up again. Or are there any other solutions?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the supplied configuration with zero-optimization stage 0, then inspect deepspeed/runtime/engine.py at line 1571 and deepspeed/runtime/fp16/fused_optimizer.py at line 255. Compare the timer setup and optimizer step behavior with stage 2; done means the stage 0 configuration no longer raises the reported NoneType callable error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100