facebookresearch / facebookresearch/fairseq2
Add VeRA support
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe:**
LoRA is designed to reduce the memory requirements to finetune LLMs and already exists in Fairseq2 to some capacity. [VeRA](https://arxiv.org/abs/2308.03303) reduces the memory overhead of [vanilla LoRA](https://arxiv.org/abs/2310.11454) even further by training a single vector that is treated as a diagonal matrix that is multiplied by two other matrices initialized using a normal distribution.
**Describe the solution you would like:**
There should be a separate class to wrap models with VeRA like there is currently with LoRA and a premade recipe would also be nice to have.
**Describe the alternatives you have considered:**
None
**Additional Context:**
None
Contributor guide
Research direction
Start by locating the existing LoRA model wrapper and premade recipe in Fairseq2, then read the linked VeRA paper to understand the requested parameterization. Done means Fairseq2 has a separate VeRA wrapper and a premade recipe, with coverage comparable to the current LoRA support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100