facebookresearch / facebookresearch/fairseq2

Add Sophia optimizer

Open
#852 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe:**
[Sophia](https://github.com/Liuhong99/Sophia) is a fairly new optimization algorithm for training language models and boasts some substantial improvements (2x speed up over Adam in wall-clock time in GPT-2 tests). Sophia seems like a nice addition to Fairseq2.

**Describe the solution you would like:**
It would be nice to offer Sophia as a finetuning recipe invoked with something like...
```
fairseq2 lm instruction_finetune --preset llama3_2_1b_instruct_sophiag
```

**Describe the alternatives you have considered:**
AdamW already exists as part of Fairseq2 as the default optimization algorithm.

**Additional Context:**
None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.