huggingface / huggingface/diffusers

Support multiple control nets in the `StableDiffusionControlNetXSPipeline`/`StableDiffusionXLControlNetXSPipeline`

Open
#8,434 12 comments 0 reactions 0 assignees View on GitHub
community-examples contributions-welcome good first issue
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

**Is your feature request related to a problem? Please describe.**
The `StableDiffusionControlNetPipeline`/`StableDiffusionXLControlNetPipeline` allow you to specify a list of control nets (i.e. depth and canny)

However, the new `ControlNetXS` pipelines only allow you provide a single control net

**Describe the solution you'd like.**
I want to be able to do:
```python
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
canny_controlnet = ControlNetXSAdapter.from_pretrained(
"UmerHA/Testing-ConrolNetXS-SDXL-canny", torch_dtype=torch.float16
)
depth_controlnet = ControlNetXSAdapter.from_pretrained(
"UmerHA/Testing-ConrolNetXS-SDXL-depth", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetXSPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", controlnet=[canny_controlnet, depth_controlnet], torch_dtype=torch.float16
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.