huggingface / huggingface/diffusers

NaN in DPMSolverMultistepInverseScheduler

Offen
#10,748 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

Hi, everyone, I'm new to diffusers. I'm trying to use DPMSolverMultistepInverseScheduler for DDIM inversion. The applied config is:
```python
dpmpp_2m_sde_karras_scheduler_inv = DPMSolverMultistepInverseScheduler(
num_train_timesteps=1000,
beta_start=0.00085,
beta_end=0.012,
algorithm_type="sde-dpmsolver++",
use_karras_sigmas=True,
steps_offset=1
)
```
And the DDIM inversion is realized through:
```python
self.scheduler.set_timesteps(self.inv_config.steps)
timesteps = self.scheduler.timesteps
with torch.autocast(device_type=self.device, dtype=self.dtype):
for i, t in enumerate(tqdm(timesteps)):
noises = []
x_index = torch.arange(len(x))
batches = x_index.split(self.batch_size, dim = 0)
for batch in batches:
noise = self.pred_noise(
x[batch], conds, timesteps[i], concat_conds=x[batch], batch_idx=batch)
noises += [noise]
noises = torch.cat(noises)

x = self.scheduler.step(noises, t, x, generator=self.rng, return_dict=False)[0]
```
But NaN occurs in the first scheduler step. I dug into it and found it happens in `dpm_solver_first_order_update`:
https://github.com/huggingface/diffusers/blob/560fb5f4d65b8593c13e4be50a59b1fd9c2d9992/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py#L627-L633
The self.sigmas is `tensor([ 0.0292, 0.0462, 0.0710, 0.1065, 0.1563, 0.2249, 0.3178, 0.4417,
0.6050, 0.8176, 1.0911, 1.4396, 1.8795, 2.4300, 3.1132, 3.9548,
4.9844, 6.2356, 7.7471, 9.5622, 11.7303, 14.3068, 17.3539, 20.9411,
25.1461, 25.1461])`. Its increasing order leads `lambda_t ` to be smaller than `lambda_s` and therefore a negative `h`.
As a result, in https://github.com/huggingface/diffusers/blob/560fb5f4d65b8593c13e4be50a59b1fd9c2d9992/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py#L638-L644
`torch.sqrt(1.0 - torch.exp(-2 * h))` becomes NaN. But I noticed that in `DPMSolverMultistepScheduler`, the problem is avoided by applying flip:
https://github.com/huggingface/diffusers/blob/560fb5f4d65b8593c13e4be50a59b1fd9c2d9992/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py#L395-L396
I've searched for many usage examples, but I still can't figure out the stem of the problem. Can anybody give a help?🙏

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne in src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py bei dpm_solver_first_order_update und verwende die gemeldete Scheduler-Konfiguration sowie NaN im ersten Schritt als Reproduktion. Vergleiche die Sigma-Reihenfolge und die Behandlung von flip mit scheduling_dpmsolver_multistep.py, insbesondere die referenzierten Zeilen. Erledigt ist es, wenn die gezeigte Inversionskonfiguration ihren ersten Scheduler-Schritt abschließt, ohne NaN zu erzeugen.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.