huggingface / huggingface/diffusers
Add IP-Adapter support to StableDiffusionXLInstructPix2PixPipeline
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
**Feature Description**
StableDiffusionXLInstructPix2PixPipeline is the only InstructPix2Pix pipeline without IP-Adapter support. The SD version got it in #7820, but the SDXL one has no IPAdapterMixin, no image_encoder / feature_extractor components, and no ip_adapter_image argument, so there's no way to combine an instruction edit with an image prompt at SDXL resolution.
**Solution I'd like to introduce**
Mirror the SD InstructPix2Pix pipeline :
- add IPAdapterMixin and the optional image_encoder / feature_extractor components
- add ip_adapter_image / ip_adapter_image_embeds to __call__ and check_inputs
- Copied from StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds, which is already three-way-CFG aware
The one design question I'd like to confirm before opening a PR is the CFG ordering. InstructPix2Pix expands the batch as [text, image, uncond] and lays out prompt_embeds as [prompt, negative, negative]. Since IP-Adapter conditioning is cross-attention conditioning like the prompt, I ordered the image embeddings the same way, so IP-Adapter strength is scaled by guidance_scale rather than image_guidance_scale. That matches what #7820 does for the SD pipeline.
**Alternatives considered**
I first ordered the embeddings [cond, cond, uncond], putting the IP-Adapter conditioning in the image_guidance_scale term instead. It runs, but it disagrees with the SD pipeline and makes the adapter much weaker at default settings, so I dropped it. Happy to revisit if you'd prefer that behaviour for the edit pipelines.
**Additional context.**
I have a working branch with tests (21 passed, 18 skipped on tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py, with check_copies / check_dummies / make quality clean). Would you take a PR for this?
Contributor guide
Research direction
Start with StableDiffusionXLInstructPix2PixPipeline and compare its structure with StableDiffusionInstructPix2PixPipeline.prepare_ip_adapter_image_embeds. Review tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_instruction_pix2pix.py and the existing SD implementation for the requested components, arguments, and CFG behavior. Done means the SDXL pipeline supports IP-Adapter image inputs or embeddings with passing pipeline and quality checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100