huggingface / huggingface/diffusers
Support Original Checkpoint-Compatible PEFT Adapters
- Lenguaje dominante
- Python
- Estrellas
- 34.5k
- Forks
- 7.3k
- Merge medio
- 3 d 3 h
- PR fusionados (30 d)
- 91
Descripción
**Is your feature request related to a problem? Please describe.**
My understanding is that `diffusers` does not currently support original checkpoint-compatible PEFT adapters aside from LoRAs.
**Describe the solution you'd like.**
I think `diffusers` should support loading general `peft`-compatible checkpoints which are compatible with the original checkpoint structure.
In particular, my proposal is that it could be supported by the following strategy:
1. Make the `diffusers` checkpoint and PEFT checkpoint weights coincide.
2. Inject the checkpoint normally using `peft`.
See https://github.com/huggingface/diffusers/pull/13861#issuecomment-4756872880 for a slightly more fleshed out version of the above.
**Describe alternatives you've considered.**
`diffusers` supports LoRA checkpoints by converting the checkpoints to the `diffusers` format. However, differences between `diffusers` checkpoints and potential LoRA checkpoints, most notably split Q,K,V projections in `diffusers` vs fused QKV projections in LoRA checkpoints, means that the adapter checkpoint weights need to be modified to make this possible. In general, if weight shape differences are present, each new PEFT technique (e.g. DoRA, IA3, etc.) would need their own technique-specific conversion logic. So, in particular, for (1) in the proposal above we should make the `diffusers` checkpoint match the original checkpoint in terms of weight shapes (for example, by fusing the QKV projections) to avoid the need for conversion logic for each PEFT technique.
**Additional context.**
Add any other context or screenshots about the feature request here.
Guía de contribución
Línea de trabajo
No se nombran archivos ni pruebas. Empieza leyendo el comentario vinculado del pull request y siguiendo las rutas existentes de conversión de checkpoints de LoRA e inyección de PEFT descritas aquí; aclara cómo deben alinearse las formas de los pesos de los checkpoints y la carga, y luego define la verificación para checkpoints compatibles con PEFT en general, sin conversión específica de la técnica.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python, pytorch
- Área
- machine-learning
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Tranquilo
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 35/100