huggingface / huggingface/diffusers

Support Original Checkpoint-Compatible PEFT Adapters

Open
#14,002 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
My understanding is that `diffusers` does not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.

**Describe the solution you'd like.**
I think `diffusers` should support loading general `peft`-compatible checkpoints which are compatible with the original checkpoint structure.

In particular, my proposal is that it could be supported by the following strategy:
1. Make the `diffusers` checkpoint and PEFT checkpoint weights coincide.
2. Inject the checkpoint normally using `peft`.

See https://github.com/huggingface/diffusers/pull/13861#issuecomment-4756872880 for a slightly more fleshed out version of the above.

**Describe alternatives you've considered.**
`diffusers` supports LoRA checkpoints by converting the checkpoints to the `diffusers` format. However, differences between `diffusers` checkpoints and potential LoRA checkpoints, most notably split Q,K,V projections in `diffusers` vs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make the `diffusers` checkpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.

**Additional context.**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reading the linked pull request comment and tracing the existing LoRA checkpoint conversion and PEFT injection paths described here; clarify how checkpoint weight shapes and loading should align, then define verification for general PEFT-compatible checkpoints without technique-specific conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.