huggingface / huggingface/diffusers
[Community] Add MagicTailor Personalization Training Script
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
### Model/Pipeline/Scheduler description
Recent advancements in fine-tuning techniques for text-to-image (T2I) personalization still struggle to distill visual concepts from reference images when there are both image-wide and spatially localized concepts present in each reference image.
The techniques in this paper are designed to improve component-controllable personalization, a novel task that pushes the boundaries of T2I models by allowing users to reconfigure specific components when personalizing visual concepts. This task is particularly challenging due to two primary obstacles: semantic pollution, where unwanted visual elements corrupt spatially localized concepts, and semantic imbalance, which causes disproportionate learning between the custom image-wide concept and spatially localized component concepts.
To overcome these challenges, MagicTailor leverages Dynamic Masked Degradation (DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual semantics.
### Open source status
- [X] The model implementation is available.
- [ ] The model weights are available (Only relevant if addition is not a scheduler).
### Provide useful links for the implementation
Paper: https://arxiv.org/pdf/2410.13370
Project Website: https://correr-zhou.github.io/MagicTailor/
Code: https://github.com/correr-zhou/MagicTailor
Contact: @Correr-Zhou
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with the MagicTailor paper and project website, then inspect the linked implementation to understand the required training pipeline. Compare its components with the repository's existing training entry points; done means a usable MagicTailor personalization training script is integrated, with the unavailable model weights treated as an open limitation.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, pytorch
- Ambito
- computer-vision, machine-learning
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 20/100