kohya-ss / kohya-ss/sd-scripts

Is alpha calculated correctly?

Open
#2,204 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

https://github.com/kohya-ss/sd-scripts/blob/f5d44fd4874ef19dfbd764158ad29187184c6fd0/networks/lora_flux.py#L1086-L1134

With `split_qkv`, we merge ModuleList into a single fused tensor. The rank of this tensor will be three or four times larger than regular. At the same time, we leave alpha unchanged. If the inference code does not take this feature into account and calculates scale using the usual formula: scale=alpha/lora_down.shape[0], then the contribution of these layers is weakened by 3-4 times.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with networks/lora_flux.py lines 1086-1134 and trace how split_qkv merges the ModuleList and sets the fused tensor's rank and alpha. Then inspect the inference scaling path to determine whether it uses the fused rank; done means confirming the scale is correct for these layers and documenting or testing the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.