huggingface / huggingface/diffusers
When use UniPCMultistepScheduler as the ODE solver, and input different num_inference_steps in StableDiffusionXLPipeline, the first inference step is wrong
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
### Describe the bug
When the user calls the StableDiffusionXLPipeline using the ckpt from stabilityai/stable-diffusion-xl-base-1.0 and sets UniPCMultistepScheduler as the inference schedule, if the user only inputs num_inference_steps without setting timesteps, the inference time schedule would be incorrect. Setting different num_inference_steps will lead to different first step input time labels, even if in all of the situations, the first step input x is pure Gaussian noise.
### Reproduction
DTYPE = torch.float16 # torch.float16 works as well, but pictures seem to be a bit worse
device = "cuda"
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",torch_dtype=DTYPE)
scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.scheduler = scheduler
pipe.to(device)
pipe.scheduler.set_timesteps(10)
print(pipe.scheduler.timesteps)
pipe.scheduler.set_timesteps(15)
print(pipe.scheduler.timesteps)
""""
In diffusers\src\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py,
Once the timestep been set by retrieve_timesteps in line 1103
, the inference timesteps are not updated to add more elements
, then, in the line 1193 the inference loop start
, and the u_net start feedforward in line 1212.
The u_net will input different first timesteps when the user set different inference nums.
"""
### Logs
```shell
The output log of the code in the reproduction section is:
tensor([901, 811, 721, 631, 541, 451, 361, 271, 181, 91])
tensor([931, 869, 807, 745, 683, 621, 559, 497, 435, 373, 311, 249, 187, 125, 63])
```
### System Info
- 🤗 Diffusers version: 0.38.0
- Platform: Windows-10-10.0.26200-SP0
- Running on Google Colab?: No
- Python version: 3.11.15
- PyTorch version (GPU?): 2.12.0+cu132 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.18.0
- Transformers version: 5.10.2
- Accelerate version: not installed
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.8.0-rc.1
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 5060 Ti, 16311 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
### Who can help?
@yiyixuxu @sayakpaul @DN6
Contributor guide
Research direction
Reproduce the issue with StableDiffusionXLPipeline and UniPCMultistepScheduler, then read diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py around retrieve_timesteps at line 1103, the inference loop at line 1193, and the U-Net call at line 1212. Compare the schedules and first input timestep for different num_inference_steps; done means the first-step schedule behavior is correct when only that parameter is supplied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100