huggingface / huggingface/diffusers

Support Original Checkpoint-Compatible PEFT Adapters

Abierto
#14,002 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
34.5k
Forks
7.3k
Merge medio
3 d 3 h
PR fusionados (30 d)
91

Descripción

**Is your feature request related to a problem? Please describe.**
My understanding is that `diffusers` does not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.

**Describe the solution you'd like.**
I think `diffusers` should support loading general `peft`-compatible checkpoints which are compatible with the original checkpoint structure.

In particular, my proposal is that it could be supported by the following strategy:
1. Make the `diffusers` checkpoint and PEFT checkpoint weights coincide.
2. Inject the checkpoint normally using `peft`.

See https://github.com/huggingface/diffusers/pull/13861#issuecomment-4756872880 for a slightly more fleshed out version of the above.

**Describe alternatives you've considered.**
`diffusers` supports LoRA checkpoints by converting the checkpoints to the `diffusers` format. However, differences between `diffusers` checkpoints and potential LoRA checkpoints, most notably split Q,K,V projections in `diffusers` vs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make the `diffusers` checkpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.

**Additional context.**
Add any other context or screenshots about the feature request here.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

No se nombran archivos ni pruebas. Empieza leyendo el comentario vinculado del pull request y siguiendo las rutas existentes de conversión de checkpoints de LoRA e inyección de PEFT descritas aquí; aclara cómo deben alinearse las formas de los pesos de los checkpoints y la carga, y luego define la verificación para checkpoints compatibles con PEFT en general, sin conversión específica de la técnica.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python, pytorch
Área
machine-learning
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Tranquilo
Claridad
Necesita aclaración
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.