huggingface / huggingface/diffusers

Support Original Checkpoint-Compatible PEFT Adapters

Offen
#14,002 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

**Is your feature request related to a problem? Please describe.**
My understanding is that `diffusers` does not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.

**Describe the solution you'd like.**
I think `diffusers` should support loading general `peft`-compatible checkpoints which are compatible with the original checkpoint structure.

In particular, my proposal is that it could be supported by the following strategy:
1. Make the `diffusers` checkpoint and PEFT checkpoint weights coincide.
2. Inject the checkpoint normally using `peft`.

See https://github.com/huggingface/diffusers/pull/13861#issuecomment-4756872880 for a slightly more fleshed out version of the above.

**Describe alternatives you've considered.**
`diffusers` supports LoRA checkpoints by converting the checkpoints to the `diffusers` format. However, differences between `diffusers` checkpoints and potential LoRA checkpoints, most notably split Q,K,V projections in `diffusers` vs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make the `diffusers` checkpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.

**Additional context.**
Add any other context or screenshots about the feature request here.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

No files or tests are named. Start by reading the linked pull request comment and tracing the existing LoRA checkpoint conversion and PEFT injection paths described here; clarify how checkpoint weight shapes and loading should align, then define verification for general PEFT-compatible checkpoints without technique-specific conversion.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Ruhig
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.