deepseek-ai / deepseek-ai/FlashMLA
Kronecker factors instead of LoRA?
Open
- Dominant language
- C++
- Stars
- 12.9k
- Forks
- 1.2k
- Avg merge
- 4h 20m
- Merged PRs (30d)
- 2
Description
have you considered swapping the lora-style rank adapters for kronecker-factorized ones (similar to LoKr)? since the 512-wide heads reshape cleanly (to eg. 16×32 or 8×64), you can get a higher effective rank for the same param budget. it could possibly capture cross-dim structure better. just curious if you’ve benchmarked that or already ruled it out.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.