huggingface / huggingface/diffusers

Unexpected behavior of `image_processor.apply_overlay`

Abierto
#9,166 12 comentarios 0 reacciones 1 asignado Reclamado por @asomoza Ver en GitHub
bug stale
Lenguaje dominante
Python
Estrellas
34.5k
Forks
7.3k
Merge medio
3 d 3 h
PR fusionados (30 d)
91

Descripción

### Describe the bug

The `apply_overlay` method of `image_processor` works unexpectedly if `init_image` and `image` have different shapes. For example it can appear using `StableDiffusionXLInpaintPipeline` with `padding_mask_crop` value filled.

According to `apply_overlay` code, it resizes `init_image` to the `image` size and works with all images in these coordinates. And if `crop_coordinates` are in `init_image` coordinates (that is expected and intuitive, `StableDiffusionXLInpaintPipeline` uses exactly this coordinates system) then it inserts `image` into the wrong place.

### Reproduction

This code corresponds to [the example](https://huggingface.co/docs/diffusers/en/using-diffusers/inpaint#padding-mask-crop) from official documentation.

```
import torch
from diffusers import StableDiffusionXLInpaintPipeline

from diffusers.utils import load_image
import numpy as np
from PIL import Image

init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")

inpainting_mask = np.zeros_like(init_image)
inpainting_mask[210:555, 71:600] = 255
inpainting_map = Image.fromarray(inpainting_mask)

base_model_name: str = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"
pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
base_model_name,
torch_dtype=torch.float16,
variant="fp16",
).to("cuda")

image = pipe(
"cat",
image=init_image,
mask_image=inpainting_map,
strength=0.99,
num_inference_steps=20,
guidance_scale=8.5,
padding_mask_crop=16,
).images[0]

image.save("output.png")
```

Output Image
![output](https://github.com/user-attachments/assets/9da87ae9-b9fc-42fa-a361-1fc894b2e094)

As we can see, the crop was pasted into the wrong location such as `crop_coordinates` used in `apply_overlay` method are in original image system, but all was resized to the crop image size

### Logs

_No response_

### System Info

- 🤗 Diffusers version: 0.28.2
- Platform: Ubuntu 22.04.4 LTS - Linux-5.15.0-113-generic-x86_64-with-glibc2.35
- Running on a notebook?: No
- Running on Google Colab?: No
- Python version: 3.10.14
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.24.0
- Transformers version: 4.41.2
- Accelerate version: 0.22.0
- PEFT version: 0.11.1
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: 0.0.23
- Accelerator: NVIDIA RTX A5000, 24564 MiB VRAM
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.