deepspeedai / deepspeedai/DeepSpeed

[BUG] inconsistent optimizer naming and defaults

Open
#5,214 3 comments 0 reactions 1 assignee View on GitHub

@loadams is already working on this.

Since Mar 5, 2024.

  • #5382 by @loadams — open
bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
I've noticed a couple of minor inconsistencies with the Deepspeed provided optimizers.

  1. I expect CPU adam and GPU adam to have identical naming. However, CPU adam has modelparams and adamw_mode while GPU adam has params and adam_w_mode. It's a bit annoying that the same instantiation code doesn't work for both optimizers.
  2. The default weight decay on both optimizers is set to 0, which is confusing given that the default optimizer is set to use AdamW mode (and I think most practitioners do as well). My understanding is that AdamW with 0 weight decay is equivalent to Adam, and probably not what practitioners intended to use. The PyTorch default uses 0.01 for weight decay, and the docs claim

apex.optimizers.FusedAdam may be used as a drop-in replacement for torch.optim.AdamW, or torch.optim.Adam with adam_w_mode=False:

One option to maintain backwards compatibility is to create new AdamW optimizers for CPU and GPU, each of which just sets weight decay 0.01 by default. This would also match the torch implementation, which has Adam w/ weight decay = 0 and AdamW w/weight decay=0.01.

To Reproduce
Deepspeed docs: https://deepspeed.readthedocs.io/en/latest/optimizers.html
Torch adamw docs: https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html

Expected behavior

  • Arguments to CPU and GPU adam are identical
  • Default parameters match those of Pytorch's

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.