huggingface / huggingface/diffusers
When use UniPCMultistepScheduler as the ODE solver, and input different num_inference_steps in StableDiffusionXLPipeline, the first inference step is wrong
- Lingua principale
- Python
- Stelle
- 34.5k
- Fork
- 7.3k
- Merge medio
- 3g 3h
- PR unite (30g)
- 91
Descrizione
### Describe the bug
When the user calls the StableDiffusionXLPipeline using the ckpt from stabilityai/stable-diffusion-xl-base-1.0 and sets UniPCMultistepScheduler as the inference schedule, if the user only inputs num_inference_steps without setting timesteps, the inference time schedule would be incorrect. Setting different num_inference_steps will lead to different first step input time labels, even if in all of the situations, the first step input x is pure Gaussian noise.
### Reproduction
DTYPE = torch.float16 # torch.float16 works as well, but pictures seem to be a bit worse
device = "cuda"
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",torch_dtype=DTYPE)
scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.scheduler = scheduler
pipe.to(device)
pipe.scheduler.set_timesteps(10)
print(pipe.scheduler.timesteps)
pipe.scheduler.set_timesteps(15)
print(pipe.scheduler.timesteps)
""""
In diffusers\src\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py,
Once the timestep been set by retrieve_timesteps in line 1103
, the inference timesteps are not updated to add more elements
, then, in the line 1193 the inference loop start
, and the u_net start feedforward in line 1212.
The u_net will input different first timesteps when the user set different inference nums.
"""
### Logs
```shell
The output log of the code in the reproduction section is:
tensor([901, 811, 721, 631, 541, 451, 361, 271, 181, 91])
tensor([931, 869, 807, 745, 683, 621, 559, 497, 435, 373, 311, 249, 187, 125, 63])
```
### System Info
- 🤗 Diffusers version: 0.38.0
- Platform: Windows-10-10.0.26200-SP0
- Running on Google Colab?: No
- Python version: 3.11.15
- PyTorch version (GPU?): 2.12.0+cu132 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.18.0
- Transformers version: 5.10.2
- Accelerate version: not installed
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.8.0-rc.1
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 5060 Ti, 16311 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
### Who can help?
@yiyixuxu @sayakpaul @DN6
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Riprodurre il problema con StableDiffusionXLPipeline e UniPCMultistepScheduler, quindi leggere diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py intorno a retrieve_timesteps alla riga 1103, il ciclo di inferenza alla riga 1193 e la chiamata a U-Net alla riga 1212. Confrontare gli schedule e il primo timestep di input per diversi num_inference_steps; il lavoro è completato quando il comportamento dello schedule al primo passaggio è corretto fornendo solo quel parametro.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, pytorch
- Ambito
- machine-learning
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100