kohya-ss / kohya-ss/sd-scripts
Is alpha calculated correctly?
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
https://github.com/kohya-ss/sd-scripts/blob/f5d44fd4874ef19dfbd764158ad29187184c6fd0/networks/lora_flux.py#L1086-L1134
With `split_qkv`, we merge ModuleList into a single fused tensor. The rank of this tensor will be three or four times larger than regular. At the same time, we leave alpha unchanged. If the inference code does not take this feature into account and calculates scale using the usual formula: scale=alpha/lora_down.shape[0], then the contribution of these layers is weakened by 3-4 times.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with networks/lora_flux.py lines 1086-1134 and trace how split_qkv merges the ModuleList and sets the fused tensor's rank and alpha. Then inspect the inference scaling path to determine whether it uses the fused rank; done means confirming the scale is correct for these layers and documenting or testing the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100