huggingface / huggingface/diffusers

Empty timesteps in StableDiffusionXLImg2ImgPipeline with SDXL-Turbo using strength > 0.5 and 1 inference step

Aperta
#11,092 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

### Describe the bug

When using StableDiffusionXLImg2ImgPipeline with the SDXL-Turbo model, setting num_inference_steps=1 and certain strength values (typically > 0.5), the pipeline crashes with a tensor error during generation. The issue occurs specifically in the get_timesteps method, where an empty tensor is created due to how the timestep calculation is performed.
The root cause is in the get_timesteps method in pipeline_stable_diffusion_xl_img2img.py. For certain combinations of strength and num_inference_steps, the method returns an empty list of timesteps which causes the latent tensor to become empty later in the pipeline.

**Current Behavior**
The get_timesteps method calculates:

init_timestep = min(int(num_inference_steps * strength), num_inference_steps)
t_start = max(num_inference_steps - init_timestep, 0)
Then retrieves timesteps: timesteps = self.scheduler.timesteps[t_start * self.scheduler.order:]

When num_inference_steps=1 and strength=0.8:

init_timestep = min(int(1 * 0.8), 1) = 0
t_start = max(1 - 0, 0) = 1
Resulting in timesteps = scheduler.timesteps[1 * scheduler.order:] which is an empty slice

This empty tensor later causes a failure in the add_noise method when trying to apply noise to the latents.

**Expected Behavior**
The pipeline should ensure that at least one timestep is available regardless of the strength and inference step settings. A patch could include a check to ensure the timestep slice is never empty, falling back to a single timestep if necessary.

**Workaround**
A temporary workaround is to monkey-patch the get_timesteps method:

```
import types

# Fix the get_timesteps method
original_get_timesteps = pipe.get_timesteps

def fixed_get_timesteps(self, num_inference_steps, strength, device, denoising_start=None):
if denoising_start is None:
init_timestep = min(int(num_inference_steps * strength), num_inference_steps)
t_start = max(num_inference_steps - init_timestep, 0)

# Check if we'd get an empty slice and adjust if needed
if t_start * self.scheduler.order >= len(self.scheduler.timesteps):
t_start = 0 # Fallback to using all timesteps

timesteps = self.scheduler.timesteps[t_start * self.scheduler.order:]

# Extra safety check
if len(timesteps) == 0:
timesteps = self.scheduler.timesteps[0:1] # Always use at least one timestep

if hasattr(self.scheduler, "set_begin_index"):
self.scheduler.set_begin_index(t_start * self.scheduler.order)

return timesteps, num_inference_steps - t_start
else:
return original_get_timesteps(num_inference_steps, strength, device, denoising_start)

# Apply the patch
pipe.get_timesteps = types.MethodType(fixed_get_timesteps, pipe)
This issue is particularly significant for users of SDXL-Turbo, which is designed for ultra-fast generation with just 1-4 inference steps.
```

### Reproduction

```
import torch
from diffusers import StableDiffusionXLImg2ImgPipeline
from diffusers.utils import load_image

# Load model
pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
"stabilityai/sdxl-turbo",
torch_dtype=torch.float16
).to("mps") # This can be any device: cpu, cuda, mps

# Load IP-Adapter (this is not required to trigger the bug, but shows a common use case)
pipe.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.safetensors"
)

# Load input image
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png").resize((512, 512))

# This will trigger the bug
try:
output = pipe(
prompt="high quality, best quality",
image=image,
strength=0.8, # Values > 0.5 with 1 step often trigger the issue
num_inference_steps=1,
guidance_scale=1.0
)
except Exception as e:
print(f"Error: {e}")
# The error occurs during the pipeline run, either as an empty tensor error
# or down the pipeline when it tries to reshape a tensor with 0 elements
```

### Logs

```shell
.../.venv/bin/python .../ipadapter/tester.py
Loading pipeline components...: 100%|██████████| 7/7 [00:14<00:00, 2.04s/it]
0it [00:00, ?it/s]
Traceback (most recent call last):
File ".../ipadapter/tester.py", line 129, in
new_image = pipeline(
^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py", line 1478, in __call__
image = self.vae.decode(latents, return_dict=False)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/autoencoder_kl.py", line 327, in decode
decoded = self._decode(z).sample
^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/autoencoder_kl.py", line 298, in _decode
dec = self.decoder(z)
^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/vae.py", line 345, in forward
sample = self.mid_block(sample, latent_embeds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 761, in forward
hidden_states = attn(hidden_states, temb=temb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/models/attention_processor.py", line 588, in forward
return self.processor(
^^^^^^^^^^^^^^^
File ".../.venv/lib/python3.12/site-packages/diffusers/models/attention_processor.py", line 3274, in __call__
query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1, 512] because the unspecified dimension size -1 can be any value and is ambiguous

Process finished with exit code 1
```

### System Info

diffusers version: 0.24.0
Platform: macOS
Python version: 3.12
PyTorch version (GPU?): 2.1.0 (MPS)
Using GPU in script?: Yes (Apple MPS)
Using distributed or parallel set-up in script?: No

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.