huggingface / huggingface/diffusers
Fused QKV projections incompatible with training
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
### Describe the bug
I've enabled fused qkv projections in SimpleTuner, but it took quite a bit of investigation and effort.
1. any PEFT LoRAs become fused as well. we have to adjust the lora_target to include `to_qkv` instead of the split target layer names.
2. the `fuse_qkv_projections` method on the Attention class does not delete the original qkv split layers, wasting VRAM
- the remaining qkv split layers can be inadvertently used for training, targeted by PEFT LoRA or EMA Model
4. the `unfuse_qkv_projections` method on the Attention class actually doesn't do what one would expect, the weights aren't copied back from the fused layer into the splits. it merely just marks the fusion as disabled and swaps attn processors back.
5. EMAModel actually works perfectly fine with fused QKV projection training, it requires no modification.
### Reproduction
The concept is a bit complex for supplying a minimal reproducer, but we can select individual aspects of this issue report to create new, more specific issue reports for, containing all of the relevant info needed to recreate.
### Logs
```shell
```
### System Info
Latest Diffusers main.
### Who can help?
@a-r-r-o-w @DN6 @sayakpaul
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia ispezionando i metodi fuse_qkv_projections e unfuse_qkv_projections della classe Attention, quindi traccia come i target PEFT LoRA e EMA Model interagiscono con i layer fusi e separati. Riproduci i singoli comportamenti descritti nel report invece dello scenario completo. Il lavoro è completato quando training, targeting di LoRA, pulizia dei layer e unfusion si comportano in modo coerente con i requisiti dell’issue.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, pytorch
- Ambito
- machine-learning
- Tipo di issue
- Bug
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 28/100