deepspeedai / deepspeedai/DeepSpeed
[BUG] inconsistent optimizer naming and defaults
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
I've noticed a couple of minor inconsistencies with the Deepspeed provided optimizers.
- I expect CPU adam and GPU adam to have identical naming. However, CPU adam has
modelparamsandadamw_modewhile GPU adam hasparamsandadam_w_mode. It's a bit annoying that the same instantiation code doesn't work for both optimizers. - The default weight decay on both optimizers is set to 0, which is confusing given that the default optimizer is set to use AdamW mode (and I think most practitioners do as well). My understanding is that AdamW with 0 weight decay is equivalent to Adam, and probably not what practitioners intended to use. The PyTorch default uses 0.01 for weight decay, and the docs claim
apex.optimizers.FusedAdam may be used as a drop-in replacement for torch.optim.AdamW, or torch.optim.Adam with adam_w_mode=False:
One option to maintain backwards compatibility is to create new AdamW optimizers for CPU and GPU, each of which just sets weight decay 0.01 by default. This would also match the torch implementation, which has Adam w/ weight decay = 0 and AdamW w/weight decay=0.01.
To Reproduce
Deepspeed docs: https://deepspeed.readthedocs.io/en/latest/optimizers.html
Torch adamw docs: https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
Expected behavior
- Arguments to CPU and GPU adam are identical
- Default parameters match those of Pytorch's
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.