huggingface / huggingface/diffusers

Add Conditional Diffusion Distillation

Offen
#8,309 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Model/Pipeline/Scheduler description

Conditional Diffusion Distillation (CoDi) is a new diffusion generation method recently proposed by Google Research and Johns Hopkins University. Accepted by CVPR24, CoDi is based on consistency models and offers a significant advancement in accelerating latent diffusion models. This method enables faster generation in just 1-4 steps.

Key Features:

- Parameter-Efficient Distillation: CoDi is the first method that allows users to accelerate any diffusion model by simply loading a pre-trained acceleration ControlNet.
- No Architectural Changes Required: The process does not require modifications to the diffusion scheduler or model architecture, ensuring seamless integration.
- Enhanced Performance: For example, models like `stablediffusionapi/juggernaut-reborn` can be accelerated to generate results in 4 steps without the need for distillation of the juggernaut-reborn model.

![image_1](https://github.com/huggingface/diffusers/assets/13622651/359544a7-8ad6-4827-87af-bdf3790dc144)

The difference between Conditional Diffusion Distillation and recent LCM-LORA is listed below

| | Conditional Diffusion Distillation (CoDi) | LCM-LORA |
|---|---|---|
| Scheduler | Anything (Euler is tested) | LCM |
| Adapter | ControlNet | LORA |
| Full-training | Available | None |
| Backbone| SD1.5 (including its variant like juggernaut-reborn) | SD and SXL |

### Open source status

- [X] The model implementation is available.
- [X] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

project page: https://fast-codi.github.io
paper: https://arxiv.org/abs/2310.01407

@MKFMIKU will submit a PR for providing the trainng code in PyTorch and a rough pretrained model.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with the CoDi project page and paper, then review the existing diffusion model, pipeline, ControlNet, and scheduler integrations in diffusers. Add support for the described pretrained acceleration ControlNet without requiring scheduler or architecture changes, and verify generation in the stated 1–4 steps.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.