huggingface / huggingface/diffusers
Improve SD35 LoRA support to cover most popular LoRA formats
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
SD3.x pipeline does implement `SD3LoraLoaderMixin` and as such `load_lora_weights` on SD3.x does "work".
However, attempting to load any of the most popular LoRAs results in silent failure:
load is successful without any warnings, but loads ZERO keys.
Looking at implementation at:
Shows that if there are no matching keys in `text_encoder_state_dict`, `text_encoder_state_dict`, `text_encoder_2_state_dict`, there is no warning raised, the method simply does NOTHING.
When looking at `state_dict` of loaded LoRA, it shows that keys are NOT in the expected format - they need remapping.
For example:
```dict_keys(['lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.alpha', 'lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.lora_down.weight', 'lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.lora_up.weight', 'lora_unet_joint_blocks_0_context_block_attn_proj.alpha' ...])```
Below are download links for some of the popular LoRAs for SD35, all experience same behavior:
-
-
-
CC @yiyixuxu @sayakpaul @DN6 @asomoza
CC @AI-Casanova
Contributor guide
Assessment
This issue has not been assessed yet.