facebookresearch / facebookresearch/fairseq2

Add LoRA-FA support

Open
#859 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe:**
LoRA is designed to reduce the memory requirements to finetune LLMs and already exists in Fairseq2 to some capacity. [LoRA-FA](https://arxiv.org/abs/2308.03303) reduces the memory overhead of [vanilla LoRA](https://arxiv.org/abs/2106.09685) even further by freezing the A matrix while leaving the B matrix as trainable.

**Describe the solution you would like:**
Because LoRA-FA is effectively an extension of LoRA, there should be an additional parameter in the LoRAConfig class that is used to effectively enable/disable LoRa-FA.

**Describe the alternatives you have considered:**
None

**Additional Context:**
None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.