huggingface / huggingface/diffusers

Unexpected behavior of `image_processor.apply_overlay`

Ouverte
#9,166 12 commentaires 0 réactions 1 personne assignée Réclamée par @asomoza Voir sur GitHub
bug stale
Langage dominant
Python
Étoiles
34.5k
Forks
7.3k
Merge moyen
3 j 3 h
PR mergées (30 j)
91

Description

### Describe the bug

The `apply_overlay` method of `image_processor` works unexpectedly if `init_image` and `image` have different shapes. For example it can appear using `StableDiffusionXLInpaintPipeline` with `padding_mask_crop` value filled.

According to `apply_overlay` code, it resizes `init_image` to the `image` size and works with all images in these coordinates. And if `crop_coordinates` are in `init_image` coordinates (that is expected and intuitive, `StableDiffusionXLInpaintPipeline` uses exactly this coordinates system) then it inserts `image` into the wrong place.

### Reproduction

This code corresponds to [the example](https://huggingface.co/docs/diffusers/en/using-diffusers/inpaint#padding-mask-crop) from official documentation.

```
import torch
from diffusers import StableDiffusionXLInpaintPipeline

from diffusers.utils import load_image
import numpy as np
from PIL import Image

init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")

inpainting_mask = np.zeros_like(init_image)
inpainting_mask[210:555, 71:600] = 255
inpainting_map = Image.fromarray(inpainting_mask)

base_model_name: str = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"
pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
base_model_name,
torch_dtype=torch.float16,
variant="fp16",
).to("cuda")

image = pipe(
"cat",
image=init_image,
mask_image=inpainting_map,
strength=0.99,
num_inference_steps=20,
guidance_scale=8.5,
padding_mask_crop=16,
).images[0]

image.save("output.png")
```

Output Image
![output](https://github.com/user-attachments/assets/9da87ae9-b9fc-42fa-a361-1fc894b2e094)

As we can see, the crop was pasted into the wrong location such as `crop_coordinates` used in `apply_overlay` method are in original image system, but all was resized to the crop image size

### Logs

_No response_

### System Info

- 🤗 Diffusers version: 0.28.2
- Platform: Ubuntu 22.04.4 LTS - Linux-5.15.0-113-generic-x86_64-with-glibc2.35
- Running on a notebook?: No
- Running on Google Colab?: No
- Python version: 3.10.14
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.24.0
- Transformers version: 4.41.2
- Accelerate version: 0.22.0
- PEFT version: 0.11.1
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: 0.0.23
- Accelerator: NVIDIA RTX A5000, 24564 MiB VRAM
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.