huggingface / huggingface/diffusers

Support Original Checkpoint-Compatible PEFT Adapters

Aperta
#14,002 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

**Is your feature request related to a problem? Please describe.**
My understanding is that `diffusers` does not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.

**Describe the solution you'd like.**
I think `diffusers` should support loading general `peft`-compatible checkpoints which are compatible with the original checkpoint structure.

In particular, my proposal is that it could be supported by the following strategy:
1. Make the `diffusers` checkpoint and PEFT checkpoint weights coincide.
2. Inject the checkpoint normally using `peft`.

See https://github.com/huggingface/diffusers/pull/13861#issuecomment-4756872880 for a slightly more fleshed out version of the above.

**Describe alternatives you've considered.**
`diffusers` supports LoRA checkpoints by converting the checkpoints to the `diffusers` format. However, differences between `diffusers` checkpoints and potential LoRA checkpoints, most notably split Q,K,V projections in `diffusers` vs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make the `diffusers` checkpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.

**Additional context.**
Add any other context or screenshots about the feature request here.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

No files or tests are named. Start by reading the linked pull request comment and tracing the existing LoRA checkpoint conversion and PEFT injection paths described here; clarify how checkpoint weight shapes and loading should align, then define verification for general PEFT-compatible checkpoints without technique-specific conversion.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Tranquilla
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.