facebookresearch / facebookresearch/fairseq2
Add LoRA+ support
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe:**
LoRA is designed to reduce the memory requirements to finetune LLMs and already exists in Fairseq2 to some capacity. [LoRA+](https://arxiv.org/abs/2402.12354) improves the speed of [vanilla LoRA](https://arxiv.org/abs/2106.09685) by setting two different learning rates for the A and B matricies and would be a nice addition to Fairseq2.
**Describe the solution you would like:**
Because LoRA+ is effectively an extension of LoRA, there should be an additional parameter in the LoRAConfig class that is used to effectively enable/disable LoRa+.
**Describe the alternatives you have considered:**
None
**Additional Context:**
None
Contributor guide
Assessment
This issue has not been assessed yet.