facebookresearch / facebookresearch/fairseq2
Add LoRA-FA support
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe:**
LoRA is designed to reduce the memory requirements to finetune LLMs and already exists in Fairseq2 to some capacity. [LoRA-FA](https://arxiv.org/abs/2308.03303) reduces the memory overhead of [vanilla LoRA](https://arxiv.org/abs/2106.09685) even further by freezing the A matrix while leaving the B matrix as trainable.
**Describe the solution you would like:**
Because LoRA-FA is effectively an extension of LoRA, there should be an additional parameter in the LoRAConfig class that is used to effectively enable/disable LoRa-FA.
**Describe the alternatives you have considered:**
None
**Additional Context:**
None
Contributor guide
Assessment
This issue has not been assessed yet.