huggingface / huggingface/optimum-graphcore
Allow setting betas parameters for LAMB optimizer
Open
enhancement
- Dominant language
- Python
- Stars
- 87
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
[From AlexC in GC]
Currently it is only possible to set betas for ADAMW optimizer. We would like to do it also for LAMB.
- Rename training argument from `adam_beta1` and `adam_beta2` to `optimizer_beta1` and `optimizer_beta2`
- In IPUTrainer.create_optimizer pass the value of the above parameters to `betas` as part of `optimizer_kwargs`
Bonus point: while you are at it you could rename the parameter `adam_epsilon` to `optimizer_epsilon`, since this value is also used by LAMB.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.