huggingface / huggingface/diffusers

Add IP-Adapter support to StableDiffusionXLInstructPix2PixPipeline

Open
#14,761 2 comments 0 reactions 0 assignees View on GitHub
feature-request pipelines
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Feature Description**
StableDiffusionXLInstructPix2PixPipeline is the only InstructPix2Pix pipeline without IP-Adapter support. The SD version got it in #7820, but the SDXL one has no IPAdapterMixin, no image_encoder / feature_extractor components, and no ip_adapter_image argument, so there's no way to combine an instruction edit with an image prompt at SDXL resolution.

**Solution I'd like to introduce**
Mirror the SD InstructPix2Pix pipeline :
- add IPAdapterMixin and the optional image_encoder / feature_extractor components
- add ip_adapter_image / ip_adapter_image_embeds to __call__ and check_inputs
- Copied from StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds, which is already three-way-CFG aware

The one design question I'd like to confirm before opening a PR is the CFG ordering. InstructPix2Pix expands the batch as [text, image, uncond] and lays out prompt_embeds as [prompt, negative, negative]. Since IP-Adapter conditioning is cross-attention conditioning like the prompt, I ordered the image embeddings the same way, so IP-Adapter strength is scaled by guidance_scale rather than image_guidance_scale. That matches what #7820 does for the SD pipeline.

**Alternatives considered**
I first ordered the embeddings [cond, cond, uncond], putting the IP-Adapter conditioning in the image_guidance_scale term instead. It runs, but it disagrees with the SD pipeline and makes the adapter much weaker at default settings, so I dropped it. Happy to revisit if you'd prefer that behaviour for the edit pipelines.

**Additional context.**
I have a working branch with tests (21 passed, 18 skipped on tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py, with check_copies / check_dummies / make quality clean). Would you take a PR for this?

Contributor guide

Open the contributing guide

Research direction

Start with StableDiffusionXLInstructPix2PixPipeline and compare its structure with StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds. Review tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py and the existing SD implementation for the requested components, arguments, and CFG behavior. Done means the SDXL pipeline supports IP-Adapter image inputs or embeddings with passing pipeline and quality checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.