kohya-ss / kohya-ss/sd-scripts

[Feature Request] Implement ZipLoRA Merge Algorithm

Open
#1,001 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

ZipLoRA, a novel method for merging low-rank adaptations (LoRAs) to achieve high-fidelity, concept-driven personalization in generative models, has shown promising results in maintaining style and subject integrity. I propose integrating this algorithm into sd-scripts to enhance its capabilities in generating personalized stylizations.

ZipLoRA leverages a "zipper-like" approach to blend two LoRAs, minimizing similar-direction sums and preserving original content and style properties. This is based on observations from recent generative model developments and the sparsity of LoRA weight matrices.

**Implementation References:**

- **Diffusers Implementation**: https://github.com/mkshing/ziplora-pytorch
- **Paper Page**: https://ziplora.github.io/

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked Diffusers implementation at github.com/mkshing/ziplora-pytorch and the ZipLoRA paper page to understand the algorithm and its expected inputs. Then locate the LoRA merging entry point in sd-scripts. Done means the algorithm is integrated into sd-scripts and can merge two LoRAs while preserving the stated style and subject properties.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.