kohya-ss / kohya-ss/sd-scripts

Separate Learning Rates for UNET and TE with Prodigy

Open
#1,926 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Looking at Tensorboard graphs I noticed that Prodigy uses the same LR for both UNET and TE. Usually it's recommended to use lower LR for TE for better results (When I'm training LORAs with non-adaptive optimizers I usually use 1/10th or 1/5th of UNET's LR). In my personal experience of training LORAs for Illustrious-XL sometimes I've had results where LORA trained with Prodigy felt too well trained.
I looked at Prodigy's github repo and have found this Issue - [Lowering TE or Unet average only](https://github.com/konstmish/prodigy/issues/16), where @adefazio mentions that his [pull request](https://github.com/konstmish/prodigy/pull/9) for Prodigy - _"supports different LR values for different layers via setting the layer_scale for each in the param group:"_
Unfortunately, I don't have necessary knowledge to integrate these changes into sd-scripts to test if it improves training results.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing Prodigy's linked issue 16 and pull request 9 to understand the proposed per-parameter-group scaling. Then identify where sd-scripts configures Prodigy for UNET and TE training; done means those components can use separate learning rates and the training behavior can be evaluated against the reported concern.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.