kohya-ss / kohya-ss/sd-scripts
Padding LoRA ranks to enable merging
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I'm quite shaky on the math, but would it be possible to pad a lower rank LoRA (with zeros, or something else) in order to merge a lower rank LoRA with a higher rank?
I've been playing with SVD distillation at 256 rank, and might want to merge in a trained LoRA, but don't have the time or resources to fully train a high rank LoRA.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are identified in the issue. First trace how LoRA ranks and merging are handled, then establish whether zero-padding preserves the intended result when combining different ranks. Done would require a decided approach and a demonstrated merge for the stated lower- and higher-rank cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100