Implementation of LAMB optimizer
Open
@Atharva-Phatak is already working on this.
Since Oct 31, 2022.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
🚀 The feature
#6323 Mentions that torchvision is looking to implement LAMB optimizer.
@datumbox I would very much like to take this issue and create a PR.
Motivation, pitch
LAMB optimizer was created because LARS optimizer performed poorly on model with attention mechanism (mainly). LAMB has shown to achieve very good performance gains across various tasks and I believe that it should be implemented in torchvision.
Alternatives
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.