huggingface / huggingface/diffusers

Fused QKV projections incompatible with training

Offen
#11,903 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Describe the bug

I've enabled fused qkv projections in SimpleTuner, but it took quite a bit of investigation and effort.

1. any PEFT LoRAs become fused as well. we have to adjust the lora_target to include `to_qkv` instead of the split target layer names.
2. the `fuse_qkv_projections` method on the Attention class does not delete the original qkv split layers, wasting VRAM
- the remaining qkv split layers can be inadvertently used for training, targeted by PEFT LoRA or EMA Model
4. the `unfuse_qkv_projections` method on the Attention class actually doesn't do what one would expect, the weights aren't copied back from the fused layer into the splits. it merely just marks the fusion as disabled and swaps attn processors back.
5. EMAModel actually works perfectly fine with fused QKV projection training, it requires no modification.

### Reproduction

The concept is a bit complex for supplying a minimal reproducer, but we can select individual aspects of this issue report to create new, more specific issue reports for, containing all of the relevant info needed to recreate.

### Logs

```shell

```

### System Info

Latest Diffusers main.

### Who can help?

@a-r-r-o-w @DN6 @sayakpaul

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by inspecting the Attention class methods fuse_qkv_projections and unfuse_qkv_projections, then trace how PEFT LoRA targets and EMA Model interact with the fused and split layers. Reproduce the individual behaviors described in the report rather than the full scenario. Done means training, LoRA targeting, layer cleanup, and unfusion behave consistently with the issue’s requirements.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Bug
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
28/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.