kohya-ss / kohya-ss/sd-scripts

Padding LoRA ranks to enable merging

Open
#117 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I'm quite shaky on the math, but would it be possible to pad a lower rank LoRA (with zeros, or something else) in order to merge a lower rank LoRA with a higher rank?

I've been playing with SVD distillation at 256 rank, and might want to merge in a trained LoRA, but don't have the time or resources to fully train a high rank LoRA.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are identified in the issue. First trace how LoRA ranks and merging are handled, then establish whether zero-padding preserves the intended result when combining different ranks. Done would require a decided approach and a demonstrated merge for the stated lower- and higher-rank cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.