Lightning-AI / Lightning-AI/lit-llama
Why didn't use matrix multiplication in the implememtation of LoRA?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 517
- PR merge metrics
- No merged PRs in 30d
Description
Hello:
I wonder why here uses an `F.conv1d` instead of matrix multiplication, namely `after_A @ lora_B`? As we know the intuition of LoRA is Low-Rank Adaptation, i.e., approximating a higher-order matrix by multiplying two lower-order matrices.
https://github.com/Lightning-AI/lit-llama/blob/da71adea0970d6d950fb966d365cfb428aef8298/lit_llama/lora.py#L318-L323
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at lit_llama/lora.py lines 318-323, then compare the shown F.conv1d call with the issue's proposed after_A @ lora_B expression. Determine whether the two paths have equivalent shapes and behavior; document the rationale or identify a concrete change only if the comparison shows one is needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100