facebookresearch / facebookresearch/fairseq2

Add LoRA+ support

Open
#858 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe:**
LoRA is designed to reduce the memory requirements to finetune LLMs and already exists in Fairseq2 to some capacity. [LoRA+](https://arxiv.org/abs/2402.12354) improves the speed of [vanilla LoRA](https://arxiv.org/abs/2106.09685) by setting two different learning rates for the A and B matricies and would be a nice addition to Fairseq2.

**Describe the solution you would like:**
Because LoRA+ is effectively an extension of LoRA, there should be an additional parameter in the LoRAConfig class that is used to effectively enable/disable LoRa+.

**Describe the alternatives you have considered:**
None

**Additional Context:**
None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.