kohya-ss / kohya-ss/sd-scripts

[Feature Request] Decouple `linear1` and `linear2` Flux layers in network_args

Open
#1,613 8 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Hi,

There's a popular discussion thread that suggests training the `proj_out` (`linear2)` module of single blocks 7 and 20 for Flux LoRAs:

https://old.reddit.com/r/StableDiffusion/comments/1f523bd/good_flux_loras_can_be_less_than_45mb_128_dim/

As far as I can tell, it is not yet possible to isolate `linear2` through the sd-scripts `network_args` flag. Perhaps this is as close as it gets:

```
--network_args "train_double_block_indices=none" "train_single_block_indices=7,20" "single_mod_dim=0"
```

I propose replacing the `single_dim` layer with e.g. `single_linear1_dim` and `single_linear2_dim`. That way, we can specify `single_linear1_dim=0` to reproduce the training method outlined in the thread above.

Or is this already possible with a different set of arguments?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the network_args handling for single_dim and the single-block linear1/linear2 layers. Check whether existing arguments can isolate linear2 for the specified blocks; done means separate dimension controls are supported and the requested configuration can train only linear2.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.