huggingface / huggingface/diffusers

Improve SD35 LoRA support to cover most popular LoRA formats

Aperta
#9,950 9 commenti 1 reazione 1 assegnatario Rivendicata da @sayakpaul Vedi su GitHub
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

SD3.x pipeline does implement `SD3LoraLoaderMixin` and as such `load_lora_weights` on SD3.x does "work".

However, attempting to load any of the most popular LoRAs results in silent failure:
load is successful without any warnings, but loads ZERO keys.

Looking at implementation at:

Shows that if there are no matching keys in `text_encoder_state_dict`, `text_encoder_state_dict`, `text_encoder_2_state_dict`, there is no warning raised, the method simply does NOTHING.

When looking at `state_dict` of loaded LoRA, it shows that keys are NOT in the expected format - they need remapping.
For example:
```dict_keys(['lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.alpha', 'lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.lora_down.weight', 'lora_unet_joint_blocks_0_context_block_adaLN_modulation_1.lora_up.weight', 'lora_unet_joint_blocks_0_context_block_attn_proj.alpha' ...])```

Below are download links for some of the popular LoRAs for SD35, all experience same behavior:
-
-
-

CC @yiyixuxu @sayakpaul @DN6 @asomoza
CC @AI-Casanova

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.