huggingface / huggingface/diffusers
[Tracker] use micro-conditioning for the SDXL trainers
- Lenguaje dominante
- Python
- Estrellas
- 34.5k
- Forks
- 7.3k
- Merge medio
- 3 d 3 h
- PR fusionados (30 d)
- 91
Descripción
SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper [here](https://arxiv.org/abs/2307.01952).
Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.
Below is a list of the training scripts where we'd like to have this change incorporated:
- [ ] [DreamBooth SDXL LoRA](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py)
- [ ] [SDXL LCM-LoRA](https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_lora_sdxl.py) (it needs to be slightly refactored as done in https://github.com/huggingface/diffusers/pull/6547 by @haofanwang)
- [ ] [ControlNet SDXL](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet_sdxl.py)
- [ ] [T2I Adapter SDXL](https://github.com/huggingface/diffusers/blob/main/examples/t2i_adapter/train_t2i_adapter_sdxl.py)
- [x] [Textual inversion SDXL](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion_sdxl.py) (same as what's mentioned for SDXL LCM-LoRA above)
- [x] [Advanced SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py)
Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.
Guía de contribución
Línea de trabajo
Start by comparing the unchecked scripts examples/dreambooth/train_dreambooth_lora_sdxl.py, examples/consistency_distillation/train_lcm_distill_lora_sdxl.py, examples/controlnet/train_controlnet_sdxl.py, and examples/t2i_adapter/train_t2i_adapter_sdxl.py with examples/text_to_image/train_text_to_image_lora_sdxl.py. Implement micro-conditioning for one example at a time, then submit an example training command using 4–10 steps and verify that the selected checklist item is covered.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python, pytorch
- Área
- machine-learning
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Activo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 65/100