kohya-ss / kohya-ss/sd-scripts
Enabling dim_from_weights or loraplus_unet_lr_ratio will cause the error: "train_blocks must be single for split mode" (content updated).
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
Hi,
Today, when I was running LoRA training for the `Flux.1` model (sd-scripts on SD3's breach), the "`train_blocks must be single for split mode`" error suddenly occurred. This error had not appeared before. After reviewing the parameter settings, I finally found the cause.
```
F:\kohya_ss\venv\lib\site-packages\transformers\models\clip\modeling_clip.py:480: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:555.)
attn_output = torch.nn.functional.scaled_dot_product_attention(
Traceback (most recent call last):
File "F:\kohya_ss\sd-scripts\flux_train_network.py", line 564, in
trainer.train(args)
File "F:\kohya_ss\sd-scripts\train_network.py", line 1177, in train
noise_pred, target, timesteps, huber_c, weighting = self.get_noise_pred_and_target(
File "F:\kohya_ss\sd-scripts\flux_train_network.py", line 427, in get_noise_pred_and_target
model_pred = call_dit(
File "F:\kohya_ss\sd-scripts\flux_train_network.py", line 393, in call_dit
assert network.train_blocks == "single", "train_blocks must be single for split mode"
AssertionError: train_blocks must be single for split mode
```
The issue was that I specified both the "`network_weights`" and "`dim_from_weights`" parameters. Once I disabled the "`dim_from_weights`" parameter, everything worked fine again.
I wonder if anyone else has encountered the same issue. Could it be that `dim_from_weights` retrieves double blocks, causing the split mode mechanism to malfunction?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with flux_train_network.py, especially call_dit around the assertion at line 393, then trace how network_weights and dim_from_weights configure the network. Reproduce training with both parameters enabled and compare it with dim_from_weights disabled. Done means the valid configuration no longer triggers the assertion, while genuinely incompatible split-mode settings remain rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100