huggingface / huggingface/diffusers

[Community] Add MagicTailor Personalization Training Script

Offen
#9,923 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
contributions-welcome stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Model/Pipeline/Scheduler description

Recent advancements in fine-tuning techniques for text-to-image (T2I) personalization still struggle to distill visual concepts from reference images when there are both image-wide and spatially localized concepts present in each reference image.

The techniques in this paper are designed to improve component-controllable personalization, a novel task that pushes the boundaries of T2I models by allowing users to reconfigure specific components when personalizing visual concepts. This task is particularly challenging due to two primary obstacles: semantic pollution, where unwanted visual elements corrupt spatially localized concepts, and semantic imbalance, which causes disproportionate learning between the custom image-wide concept and spatially localized component concepts.

To overcome these challenges, MagicTailor leverages Dynamic Masked Degradation (DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual semantics.

### Open source status

- [X] The model implementation is available.
- [ ] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2410.13370
Project Website: https://correr-zhou.github.io/MagicTailor/
Code: https://github.com/correr-zhou/MagicTailor
Contact: @Correr-Zhou

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start with the MagicTailor paper and project website, then inspect the linked implementation to understand the required training pipeline. Compare its components with the repository's existing training entry points; done means a usable MagicTailor personalization training script is integrated, with the unavailable model weights treated as an open limitation.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
computer-vision, machine-learning
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
20/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.