huggingface / huggingface/diffusers

When use UniPCMultistepScheduler as the ODE solver, and input different num_inference_steps in StableDiffusionXLPipeline, the first inference step is wrong

Aperta
#13,887 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug needs-code-example pipelines schedulers
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

### Describe the bug

When the user calls the StableDiffusionXLPipeline using the ckpt from stabilityai/stable-diffusion-xl-base-1.0 and sets UniPCMultistepScheduler as the inference schedule, if the user only inputs num_inference_steps without setting timesteps, the inference time schedule would be incorrect. Setting different num_inference_steps will lead to different first step input time labels, even if in all of the situations, the first step input x is pure Gaussian noise.

### Reproduction

DTYPE = torch.float16 # torch.float16 works as well, but pictures seem to be a bit worse
device = "cuda"

pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",torch_dtype=DTYPE)
scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.scheduler = scheduler
pipe.to(device)
pipe.scheduler.set_timesteps(10)
print(pipe.scheduler.timesteps)
pipe.scheduler.set_timesteps(15)
print(pipe.scheduler.timesteps)
""""
In diffusers\src\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py,
Once the timestep been set by retrieve_timesteps in line 1103
, the inference timesteps are not updated to add more elements
, then, in the line 1193 the inference loop start
, and the u_net start feedforward in line 1212.
The u_net will input different first timesteps when the user set different inference nums.
"""

### Logs

```shell
The output log of the code in the reproduction section is:
tensor([901, 811, 721, 631, 541, 451, 361, 271, 181, 91])
tensor([931, 869, 807, 745, 683, 621, 559, 497, 435, 373, 311, 249, 187, 125, 63])
```

### System Info

- 🤗 Diffusers version: 0.38.0
- Platform: Windows-10-10.0.26200-SP0
- Running on Google Colab?: No
- Python version: 3.11.15
- PyTorch version (GPU?): 2.12.0+cu132 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.18.0
- Transformers version: 5.10.2
- Accelerate version: not installed
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.8.0-rc.1
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 5060 Ti, 16311 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

@yiyixuxu @sayakpaul @DN6

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Riprodurre il problema con StableDiffusionXLPipeline e UniPCMultistepScheduler, quindi leggere diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py intorno a retrieve_timesteps alla riga 1103, il ciclo di inferenza alla riga 1193 e la chiamata a U-Net alla riga 1212. Confrontare gli schedule e il primo timestep di input per diversi num_inference_steps; il lavoro è completato quando il comportamento dello schedule al primo passaggio è corretto fornendo solo quel parametro.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.