huggingface / huggingface/diffusers

Add Conditional Diffusion Distillation

オープン
#8,309 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
stale
主要言語
Python
スター
34.5k
フォーク
7.3k
平均マージ
3日 3時間
マージ済み PR(30日)
91

説明

### Model/Pipeline/Scheduler description

Conditional Diffusion Distillation (CoDi) is a new diffusion generation method recently proposed by Google Research and Johns Hopkins University. Accepted by CVPR24, CoDi is based on consistency models and offers a significant advancement in accelerating latent diffusion models. This method enables faster generation in just 1-4 steps.

Key Features:

- Parameter-Efficient Distillation: CoDi is the first method that allows users to accelerate any diffusion model by simply loading a pre-trained acceleration ControlNet.
- No Architectural Changes Required: The process does not require modifications to the diffusion scheduler or model architecture, ensuring seamless integration.
- Enhanced Performance: For example, models like `stablediffusionapi/juggernaut-reborn` can be accelerated to generate results in 4 steps without the need for distillation of the juggernaut-reborn model.

![image_1](https://github.com/huggingface/diffusers/assets/13622651/359544a7-8ad6-4827-87af-bdf3790dc144)

The difference between Conditional Diffusion Distillation and recent LCM-LORA is listed below

| | Conditional Diffusion Distillation (CoDi) | LCM-LORA |
|---|---|---|
| Scheduler | Anything (Euler is tested) | LCM |
| Adapter | ControlNet | LORA |
| Full-training | Available | None |
| Backbone| SD1.5 (including its variant like juggernaut-reborn) | SD and SXL |

### Open source status

- [X] The model implementation is available.
- [X] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

project page: https://fast-codi.github.io
paper: https://arxiv.org/abs/2310.01407

@MKFMIKU will submit a PR for providing the trainng code in PyTorch and a rough pretrained model.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start with the CoDi project page and paper, then review the existing diffusion model, pipeline, ControlNet, and scheduler integrations in diffusers. Add support for the described pretrained acceleration ControlNet without requiring scheduler or architecture changes, and verify generation in the stated 1–4 steps.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python, pytorch
領域
machine-learning
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。