huggingface / huggingface/diffusers

[Tracker] use micro-conditioning for the SDXL trainers

オープン
#6,586 コメント 13 件 リアクション 1 件 担当者 0 名 GitHub で見る
advanced contributions-welcome stale
主要言語
Python
スター
34.5k
フォーク
7.3k
平均マージ
3日 3時間
マージ済み PR(30日)
91

説明

SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper [here](https://arxiv.org/abs/2307.01952).

Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.

Below is a list of the training scripts where we'd like to have this change incorporated:

- [ ] [DreamBooth SDXL LoRA](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py)
- [ ] [SDXL LCM-LoRA](https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_lora_sdxl.py) (it needs to be slightly refactored as done in https://github.com/huggingface/diffusers/pull/6547 by @haofanwang)
- [ ] [ControlNet SDXL](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet_sdxl.py)
- [ ] [T2I Adapter SDXL](https://github.com/huggingface/diffusers/blob/main/examples/t2i_adapter/train_t2i_adapter_sdxl.py)
- [x] [Textual inversion SDXL](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion_sdxl.py) (same as what's mentioned for SDXL LCM-LoRA above)
- [x] [Advanced SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py)

Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start by comparing the unchecked scripts examples/dreambooth/train_dreambooth_lora_sdxl.py, examples/consistency_distillation/train_lcm_distill_lora_sdxl.py, examples/controlnet/train_controlnet_sdxl.py, and examples/t2i_adapter/train_t2i_adapter_sdxl.py with examples/text_to_image/train_text_to_image_lora_sdxl.py. Implement micro-conditioning for one example at a time, then submit an example training command using 4–10 steps and verify that the selected checklist item is covered.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python, pytorch
領域
machine-learning
issue の種類
機能追加
難易度
4/5
見積もり時間
3〜5日
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
65/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。