huggingface / huggingface/diffusers

[Tracker] use micro-conditioning for the SDXL trainers

Ouverte
#6,586 13 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
advanced contributions-welcome stale
Langage dominant
Python
Étoiles
34.5k
Forks
7.3k
Merge moyen
3 j 3 h
PR mergées (30 j)
91

Description

SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper [here](https://arxiv.org/abs/2307.01952).

Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.

Below is a list of the training scripts where we'd like to have this change incorporated:

- [ ] [DreamBooth SDXL LoRA](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py)
- [ ] [SDXL LCM-LoRA](https://github.com/huggingface/diffusers/blob/main/examples/consistency_distillation/train_lcm_distill_lora_sdxl.py) (it needs to be slightly refactored as done in https://github.com/huggingface/diffusers/pull/6547 by @haofanwang)
- [ ] [ControlNet SDXL](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet_sdxl.py)
- [ ] [T2I Adapter SDXL](https://github.com/huggingface/diffusers/blob/main/examples/t2i_adapter/train_t2i_adapter_sdxl.py)
- [x] [Textual inversion SDXL](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion_sdxl.py) (same as what's mentioned for SDXL LCM-LoRA above)
- [x] [Advanced SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py)

Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par comparer les scripts non cochés examples/dreambooth/train_dreambooth_lora_sdxl.py, examples/consistency_distillation/train_lcm_distill_lora_sdxl.py, examples/controlnet/train_controlnet_sdxl.py et examples/t2i_adapter/train_t2i_adapter_sdxl.py avec examples/text_to_image/train_text_to_image_lora_sdxl.py. Implémentez micro-conditioning sur un seul exemple à la fois, puis soumettez une commande d’entraînement d’exemple utilisant 4–10 étapes et vérifiez que l’élément de la checklist sélectionné est couvert.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python, pytorch
Domaine
machine-learning
Type d'issue
Fonctionnalité
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Active
Clarté
Plutôt claire
Accessibilité débutants
65/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.