deepseek-ai / deepseek-ai/FlashMLA

Kronecker factors instead of LoRA?

Open
#102 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
12.9k
Forks
1.2k
Avg merge
4h 20m
Merged PRs (30d)
2

Description

have you considered swapping the lora-style rank adapters for kronecker-factorized ones (similar to LoKr)? since the 512-wide heads reshape cleanly (to eg. 16×32 or 8×64), you can get a higher effective rank for the same param budget. it could possibly capture cross-dim structure better. just curious if you’ve benchmarked that or already ruled it out.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.