huggingface / huggingface/diffusers

Add IP-Adapter support to StableDiffusionXLInstructPix2PixPipeline

Aperta
#14,761 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
feature-request pipelines
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

**Feature Description**
StableDiffusionXLInstructPix2PixPipeline is the only InstructPix2Pix pipeline without IP-Adapter support. The SD version got it in #7820, but the SDXL one has no IPAdapterMixin, no image_encoder / feature_extractor components, and no ip_adapter_image argument, so there's no way to combine an instruction edit with an image prompt at SDXL resolution.

**Solution I'd like to introduce**
Mirror the SD InstructPix2Pix pipeline :
- add IPAdapterMixin and the optional image_encoder / feature_extractor components
- add ip_adapter_image / ip_adapter_image_embeds to __call__ and check_inputs
- Copied from StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds, which is already three-way-CFG aware

The one design question I'd like to confirm before opening a PR is the CFG ordering. InstructPix2Pix expands the batch as [text, image, uncond] and lays out prompt_embeds as [prompt, negative, negative]. Since IP-Adapter conditioning is cross-attention conditioning like the prompt, I ordered the image embeddings the same way, so IP-Adapter strength is scaled by guidance_scale rather than image_guidance_scale. That matches what #7820 does for the SD pipeline.

**Alternatives considered**
I first ordered the embeddings [cond, cond, uncond], putting the IP-Adapter conditioning in the image_guidance_scale term instead. It runs, but it disagrees with the SD pipeline and makes the adapter much weaker at default settings, so I dropped it. Happy to revisit if you'd prefer that behaviour for the edit pipelines.

**Additional context.**
I have a working branch with tests (21 passed, 18 skipped on tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py, with check_copies / check_dummies / make quality clean). Would you take a PR for this?

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da StableDiffusionXLInstructPix2PixPipeline e confronta la sua struttura con StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds. Esamina tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py e l’implementazione SD esistente per i componenti, gli argomenti e il comportamento CFG richiesti. Il lavoro è completato quando la pipeline SDXL supporta input di immagini o embeddings IP-Adapter con verifiche della pipeline e della qualità superate.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
38/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.