huggingface / huggingface/diffusers

deepfloyd stage 2 crashes with tensor size mismatch when input image size is not divisible by 8

Offen
#7,842 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug stale
Vorherrschende Sprache
Python
Sterne
34.5k
Forks
7.3k
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
91

Beschreibung

### Describe the bug

DeepFloyd's upstream code supports 8px-aligned inputs for stage II, which I believe the Diffusers implementation is based upon. However, it seems that for certain sizes, there is some unfortunate interaction between the hidden states and the residual hidden states.

I'm not sure if this is something fundamental to the model - if it is, we probably want to understand the conditions under which this problem occurs and provide an error to the user about an incompatible resolution.

### Reproduction

```py
from diffusers import IFSuperResolutionPipeline
import torch
from PIL import Image
import numpy as np

torch.manual_seed(42)

# Configuration for initial image and desired output
initial_width = 86 # Adjusted width to be one-fourth of 344 (approximately)
initial_height = 64 # Adjusted height to be one-fourth of 256

# Initialize your device setting based on availability
torch_device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "xpu" if torch.xpu.is_available() else "cpu"

# Create a dummy image (86x64)
dummy_image = torch.rand((3, initial_height, initial_width), dtype=torch.float32) # Random noise image
dummy_image = (dummy_image * 255).to(torch.uint8) # Convert to 8-bit format
dummy_pil_image = Image.fromarray(dummy_image.numpy().transpose(1, 2, 0)) # Convert to PIL image for compatibility
dummy_pil_image.save("dummy_input.png") # Save the initial dummy image

# Load your stage 2 pipeline
stage2_pipe = IFSuperResolutionPipeline.from_pretrained("DeepFloyd/IF-II-M-v1.0", watermarker=None, safety_checker=None, local_files_only=False).to(device=torch_device, dtype=torch.bfloat16)

# Upscale the dummy image using stage 2 of the pipeline
upscaled_image = stage2_pipe(
prompt="A simple upscaled image",
image=dummy_pil_image,
guidance_scale=5.5,
num_inference_steps=20,
width=344,
height=256
).images[0]

upscaled_image.save("upscaled_dummy_output.png")
```

### Logs

```shell
0%| | 0/20 [00:00

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start at the IFSuperResolutionPipeline entry point and reproduce the stage II run with the 86x64 input and 344x256 output described in the issue. Trace the hidden_states and res_hidden_states shapes shown in the logs, determine whether the resolution should be rejected or handled, and verify that the reproduction no longer crashes or reports an appropriate incompatibility error.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, pytorch
Bereich
computer-vision, machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.