huggingface / huggingface/diffusers

FLUX LoRA with CLIP text-encoder weights fails (empty rank -> IndexError) under transformers>=5

Open
#13,984 3 comments 0 reactions 0 assignees View on GitHub
bug lora pipelines
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Describe the bug

Loading a kohya-style FLUX LoRA that contains **CLIP text-encoder** weights (`lora_te1_*`) fails with `IndexError: list index out of range` when `transformers>=5` is installed.

The text-encoder half of the LoRA is never loaded; `pipe.load_lora_weights(...)` raises before any adapter is injected.

### Root cause

`transformers>=5` flattened `CLIPTextModel`: the `text_model.` wrapper module was removed, so `text_encoder.named_modules()` now yields names like `encoder.layers.0.self_attn.k_proj` instead of `text_model.encoder.layers.0.self_attn.k_proj`.

The kohya→diffusers conversion (`_convert_kohya_flux_lora_to_diffusers`) still produces text-encoder keys prefixed with `text_model.`, e.g. `text_encoder.text_model.encoder.layers.0.self_attn.k_proj.lora_B.weight`.

In `diffusers/loaders/lora_base.py::_load_lora_into_text_encoder`, the `rank` dict is built by matching `text_encoder.named_modules()` against the (converted, PEFT-format) state-dict keys:

```python
for name, _ in text_encoder.named_modules():
if name.endswith((".q_proj", ".k_proj", ".v_proj", ".out_proj", ".fc1", ".fc2")):
rank_key = f"{name}.lora_B.weight" # e.g. "encoder.layers.0.self_attn.k_proj.lora_B.weight"
if rank_key in state_dict: # but keys are "text_model.encoder.layers.0...."
rank[rank_key] = state_dict[rank_key].shape[1]
```

Under transformers 5 the module names no longer carry the `text_model.` prefix while the state-dict keys still do, so **nothing matches**, `rank` stays empty, and `_create_lora_config` → `get_peft_kwargs` does `r = lora_alpha = list(rank_dict.values())[0]` → `IndexError`.

### Reproduction

```python
import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
# any kohya FLUX LoRA that includes CLIP text-encoder weights (lora_te1_*)
pipe.load_lora_weights("kohya_flux_lora_with_text_encoder.safetensors")
```

Minimal isolation of the empty-rank step:

```python
from transformers import CLIPTextModel
te = CLIPTextModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder")
print(hasattr(te, "text_model")) # False on transformers>=5 (was True on 4.x)
print([n for n, _ in te.named_modules()][:6]) # ['', 'embeddings', ..., 'encoder', 'encoder.layers']
# -> module names are now un-prefixed, but converted LoRA keys are still 'text_model.encoder....'
```

### Traceback

```
File ".../diffusers/loaders/lora_base.py", line 390, in _load_lora_into_text_encoder
lora_config = _create_lora_config(state_dict, network_alphas, metadata, rank, is_unet=False)
File ".../diffusers/utils/peft_utils.py", line 355, in _create_lora_config
lora_config_kwargs = get_peft_kwargs(...)
File ".../diffusers/utils/peft_utils.py", line 158, in get_peft_kwargs
r = lora_alpha = list(rank_dict.values())[0]
IndexError: list index out of range
```

### Suggested fix

Reconcile the converted text-encoder key namespace with the model's `named_modules()` for `transformers>=5` (drop/normalize the stale `text_model.` segment), or build `rank` in a prefix-tolerant way. A clearer error than a bare `IndexError` when `rank` ends up empty would also help.

### System Info

- diffusers 0.38.0
- transformers 5.9.0
- peft 0.19.1
- torch 2.10–2.11 (+cu128), Python 3.11/3.12, Linux
- Model: black-forest-labs/FLUX.1-dev

### Who can help?

@sayakpaul @BenjaminBossan

Contributor guide

Open the contributing guide

Research direction

Start in diffusers/loaders/lora_base.py::_load_lora_into_text_encoder and trace the converted keys from _convert_kohya_flux_lora_to_diffusers. Reproduce with a kohya FLUX LoRA containing lora_te1_* weights and transformers>=5; done means the text-encoder weights load without an empty-rank IndexError and an adapter is injected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.