facebookresearch / facebookresearch/fairseq2

Add VeRA support

Open
#864 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe:**
LoRA is designed to reduce the memory requirements to finetune LLMs and already exists in Fairseq2 to some capacity. [VeRA](https://arxiv.org/abs/2308.03303) reduces the memory overhead of [vanilla LoRA](https://arxiv.org/abs/2310.11454) even further by training a single vector that is treated as a diagonal matrix that is multiplied by two other matrices initialized using a normal distribution.

**Describe the solution you would like:**
There should be a separate class to wrap models with VeRA like there is currently with LoRA and a premade recipe would also be nice to have.

**Describe the alternatives you have considered:**
None

**Additional Context:**
None

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing LoRA model wrapper and premade recipe in Fairseq2, then read the linked VeRA paper to understand the requested parameterization. Done means Fairseq2 has a separate VeRA wrapper and a premade recipe, with coverage comparable to the current LoRA support.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.