huggingface / huggingface/diffusers
Black output in DPMSolverSinglestepScheduler with 3 order
- 主要言語
- Python
- スター
- 34.5k
- フォーク
- 7.3k
- 平均マージ
- 3日 3時間
- マージ済み PR(30日)
- 91
説明
### Describe the bug
If you run 3n-1 steps(11, 14, ...) generation using `DPMSolverSinglestepScheduler` with `solver_order=3`, it will produce NaN output on final step.
I tried to debug what went wrong and found that in this case executes second order update on zero sigma:
`[singlestep_dpm_solver_second_order_update] self.step_index=10 len(self.sigmas)=12 sigma_t=tensor(0.) sigma_s0=tensor(0.3235) sigma_s1=tensor(0.5293)`
Then this zero value passed to:
`lambda_t = torch.log(alpha_t) - torch.log(sigma_t)`
Where we got -inf and it broke all calculations.
Looks like `last_sigmas_type='zero'` not handled properly, I fixed it locally by copying handling of this case from multistep scheduler:
```py
# Improve numerical stability for small number of steps
lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
(self.config.lower_order_final and len(self.timesteps) < 15)
or self.config.final_sigmas_type == "zero"
)
lower_order_second = (
(self.step_index == len(self.timesteps) - 2) and self.config.lower_order_final and len(self.timesteps) < 15
)
if lower_order_final:
order = 1
elif lower_order_second:
order = 2 # probably should min(2, self.config.solver_order)
prev_sample = self.singlestep_dpm_solver_update(self.model_outputs, sample=self.sample, order=order)
```
Upd:
Also as I can see - there no support for sde in third order update both in `DPMSolverSinglestepScheduler` and `DPMSolverMultistepScheduler` schedulers, so that with `algorithm_type='sde-dpmsolver++'` set they both throw error:
`UnboundLocalError: local variable 'x_t' referenced before assignment`
Because there no branch to handle sde algo.
### Reproduction
```py
from diffusers import StableDiffusionPipeline, DPMSolverSinglestepScheduler
import torch
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverSinglestepScheduler.from_config({**pipe.scheduler.config, "solver_order":3})
pipe = pipe.to("cuda")
prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt, num_inference_steps=11).images[0]
image.save("astronaut_rides_horse.png")
```
### Logs
_No response_
### System Info
- `diffusers` version: 0.27.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.6
- PyTorch version (GPU?): 2.2.2+cu118 (True)
- Huggingface_hub version: 0.23.1
- Transformers version: 4.41.1
- Accelerate version: 0.30.1
- xFormers version: not installed
- Using GPU in script?: Yes, RTX 3080
- Using distributed or parallel set-up in script?: ???
### Who can help?
@yiyixuxu
コントリビューションガイド
調査の方向性
DPMSolverSinglestepScheduler とその singlestep_dpm_solver_second_order_update パスから始め、続いて issue で説明されている multistep scheduler の動作と最終ステップの処理を比較します。solver_order=3 の 11 ステップのケースを再現し、最終 sigma がゼロの場合に NaN が発生しなくなっていることを確認します。また、両方の scheduler における 3 次 sde-dpmsolver++ の処理も確認します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python, pytorch
- 領域
- machine-learning
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 42/100