facebookresearch / facebookresearch/fairseq2
Add Sophia optimizer
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe:**
[Sophia](https://github.com/Liuhong99/Sophia) is a fairly new optimization algorithm for training language models and boasts some substantial improvements (2x speed up over Adam in wall-clock time in GPT-2 tests). Sophia seems like a nice addition to Fairseq2.
**Describe the solution you would like:**
It would be nice to offer Sophia as a finetuning recipe invoked with something like...
```
fairseq2 lm instruction_finetune --preset llama3_2_1b_instruct_sophiag
```
**Describe the alternatives you have considered:**
AdamW already exists as part of Fairseq2 as the default optimization algorithm.
**Additional Context:**
None
Contributor guide
Assessment
This issue has not been assessed yet.