huggingface / huggingface/diffusers
Black output in DPMSolverSinglestepScheduler with 3 order
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
### Describe the bug
If you run 3n-1 steps(11, 14, ...) generation using `DPMSolverSinglestepScheduler` with `solver_order=3`, it will produce NaN output on final step.
I tried to debug what went wrong and found that in this case executes second order update on zero sigma:
`[singlestep_dpm_solver_second_order_update] self.step_index=10 len(self.sigmas)=12 sigma_t=tensor(0.) sigma_s0=tensor(0.3235) sigma_s1=tensor(0.5293)`
Then this zero value passed to:
`lambda_t = torch.log(alpha_t) - torch.log(sigma_t)`
Where we got -inf and it broke all calculations.
Looks like `last_sigmas_type='zero'` not handled properly, I fixed it locally by copying handling of this case from multistep scheduler:
```py
# Improve numerical stability for small number of steps
lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
(self.config.lower_order_final and len(self.timesteps) < 15)
or self.config.final_sigmas_type == "zero"
)
lower_order_second = (
(self.step_index == len(self.timesteps) - 2) and self.config.lower_order_final and len(self.timesteps) < 15
)
if lower_order_final:
order = 1
elif lower_order_second:
order = 2 # probably should min(2, self.config.solver_order)
prev_sample = self.singlestep_dpm_solver_update(self.model_outputs, sample=self.sample, order=order)
```
Upd:
Also as I can see - there no support for sde in third order update both in `DPMSolverSinglestepScheduler` and `DPMSolverMultistepScheduler` schedulers, so that with `algorithm_type='sde-dpmsolver++'` set they both throw error:
`UnboundLocalError: local variable 'x_t' referenced before assignment`
Because there no branch to handle sde algo.
### Reproduction
```py
from diffusers import StableDiffusionPipeline, DPMSolverSinglestepScheduler
import torch
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverSinglestepScheduler.from_config({**pipe.scheduler.config, "solver_order":3})
pipe = pipe.to("cuda")
prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt, num_inference_steps=11).images[0]
image.save("astronaut_rides_horse.png")
```
### Logs
_No response_
### System Info
- `diffusers` version: 0.27.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.6
- PyTorch version (GPU?): 2.2.2+cu118 (True)
- Huggingface_hub version: 0.23.1
- Transformers version: 4.41.1
- Accelerate version: 0.30.1
- xFormers version: not installed
- Using GPU in script?: Yes, RTX 3080
- Using distributed or parallel set-up in script?: ???
### Who can help?
@yiyixuxu
Contributor guide
Assessment
This issue has not been assessed yet.