huggingface / huggingface/diffusers

[Tracker] use micro-conditioning for the SDXL trainers

Offen
#6,586 13 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
advanced contributions-welcome stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper [here](https://arxiv.org/abs/2307.01952).

Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.

Below is a list of the training scripts where we'd like to have this change incorporated:

- [ ] [DreamBooth SDXL LoRA](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py)
- [ ] [SDXL LCM-LoRA](https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_lora_sdxl.py) (it needs to be slightly refactored as done in https://github.com/huggingface/diffusers/pull/6547 by @haofanwang)
- [ ] [ControlNet SDXL](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet_sdxl.py)
- [ ] [T2I Adapter SDXL](https://github.com/huggingface/diffusers/blob/main/examples/t2i_adapter/train_t2i_adapter_sdxl.py)
- [x] [Textual inversion SDXL](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion_sdxl.py) (same as what's mentioned for SDXL LCM-LoRA above)
- [x] [Advanced SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py)

Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by comparing the unchecked scripts examples/dreambooth/train_dreambooth_lora_sdxl.py, examples/consistency_distillation/train_lcm_distill_lora_sdxl.py, examples/controlnet/train_controlnet_sdxl.py, and examples/t2i_adapter/train_t2i_adapter_sdxl.py with examples/text_to_image/train_text_to_image_lora_sdxl.py. Implement micro-conditioning for one example at a time, then submit an example training command using 4–10 steps and verify that the selected checklist item is covered.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
65/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.