huggingface / huggingface/diffusers

NaN in DPMSolverMultistepInverseScheduler

Aperta
#10,748 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

Hi, everyone, I'm new to diffusers. I'm trying to use DPMSolverMultistepInverseScheduler for DDIM inversion. The applied config is:
```python
dpmpp_2m_sde_karras_scheduler_inv = DPMSolverMultistepInverseScheduler(
num_train_timesteps=1000,
beta_start=0.00085,
beta_end=0.012,
algorithm_type="sde-dpmsolver++",
use_karras_sigmas=True,
steps_offset=1
)
```
And the DDIM inversion is realized through:
```python
self.scheduler.set_timesteps(self.inv_config.steps)
timesteps = self.scheduler.timesteps
with torch.autocast(device_type=self.device, dtype=self.dtype):
for i, t in enumerate(tqdm(timesteps)):
noises = []
x_index = torch.arange(len(x))
batches = x_index.split(self.batch_size, dim = 0)
for batch in batches:
noise = self.pred_noise(
x[batch], conds, timesteps[i], concat_conds=x[batch], batch_idx=batch)
noises += [noise]
noises = torch.cat(noises)

x = self.scheduler.step(noises, t, x, generator=self.rng, return_dict=False)[0]
```
But NaN occurs in the first scheduler step. I dug into it and found it happens in `dpm_solver_first_order_update`:
https://github.com/huggingface/diffusers/blob/560fb5f4d65b8593c13e4be50a59b1fd9c2d9992/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py#L627-L633
The self.sigmas is `tensor([ 0.0292, 0.0462, 0.0710, 0.1065, 0.1563, 0.2249, 0.3178, 0.4417,
0.6050, 0.8176, 1.0911, 1.4396, 1.8795, 2.4300, 3.1132, 3.9548,
4.9844, 6.2356, 7.7471, 9.5622, 11.7303, 14.3068, 17.3539, 20.9411,
25.1461, 25.1461])`. Its increasing order leads `lambda_t ` to be smaller than `lambda_s` and therefore a negative `h`.
As a result, in https://github.com/huggingface/diffusers/blob/560fb5f4d65b8593c13e4be50a59b1fd9c2d9992/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py#L638-L644
`torch.sqrt(1.0 - torch.exp(-2 * h))` becomes NaN. But I noticed that in `DPMSolverMultistepScheduler`, the problem is avoided by applying flip:
https://github.com/huggingface/diffusers/blob/560fb5f4d65b8593c13e4be50a59b1fd9c2d9992/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py#L395-L396
I've searched for many usage examples, but I still can't figure out the stem of the problem. Can anybody give a help?🙏

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia in src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py, in dpm_solver_first_order_update, usando la configurazione dello scheduler segnalata e il NaN del primo passaggio come riproduzione. Confronta l’ordinamento di sigma e la gestione di flip con scheduling_dpmsolver_multistep.py, in particolare le righe indicate. Il lavoro è completato quando la configurazione di inversione mostrata completa il primo passaggio dello scheduler senza produrre NaN.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.