kohya-ss / kohya-ss/sd-scripts

Support individual lora module level selection and settings for more granular training

Open
#2,125 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Applying a global values for the training of text encoders and unet can lead to bad training results in some cases. By allowing different learning rates, dimensions, and alphas, for individual lora modules as well as choosing which modules to train on each block, not just as a group, will allow for better training results to be achieved. Currently, I edit the lora.py file in order to achieve this but having official support for this would be great.

For example, training the text encoder self attention modules can lead to overfitting quickly. There's currently no option to disable those specific modules or adjust the learning rate for them. In an older releases until about half a year ago, there was a bug that prevented the training of these modules, which inadvertently result in loras with noticeably better fidelity. Since this bug was fixed, it was impossible to reproduce such results because these text encoder self attention modules are always trained, and there's no option to configure them.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing lora.py, especially how current global text encoder and UNet training values are applied. Map the existing module groups and determine how per-module selection, learning rates, dimensions, and alphas would be configured. Done means users can independently enable or disable modules and set those values without editing lora.py directly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.