huggingface / huggingface/diffusers
Unexpected behavior of `image_processor.apply_overlay`
- Langage dominant
- Python
- Étoiles
- 34.5k
- Forks
- 7.3k
- Merge moyen
- 3 j 3 h
- PR mergées (30 j)
- 91
Description
### Describe the bug
The `apply_overlay` method of `image_processor` works unexpectedly if `init_image` and `image` have different shapes. For example it can appear using `StableDiffusionXLInpaintPipeline` with `padding_mask_crop` value filled.
According to `apply_overlay` code, it resizes `init_image` to the `image` size and works with all images in these coordinates. And if `crop_coordinates` are in `init_image` coordinates (that is expected and intuitive, `StableDiffusionXLInpaintPipeline` uses exactly this coordinates system) then it inserts `image` into the wrong place.
### Reproduction
This code corresponds to [the example](https://huggingface.co/docs/diffusers/en/using-diffusers/inpaint#padding-mask-crop) from official documentation.
```
import torch
from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image
import numpy as np
from PIL import Image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
inpainting_mask = np.zeros_like(init_image)
inpainting_mask[210:555, 71:600] = 255
inpainting_map = Image.fromarray(inpainting_mask)
base_model_name: str = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"
pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
base_model_name,
torch_dtype=torch.float16,
variant="fp16",
).to("cuda")
image = pipe(
"cat",
image=init_image,
mask_image=inpainting_map,
strength=0.99,
num_inference_steps=20,
guidance_scale=8.5,
padding_mask_crop=16,
).images[0]
image.save("output.png")
```
Output Image

As we can see, the crop was pasted into the wrong location such as `crop_coordinates` used in `apply_overlay` method are in original image system, but all was resized to the crop image size
### Logs
_No response_
### System Info
- 🤗 Diffusers version: 0.28.2
- Platform: Ubuntu 22.04.4 LTS - Linux-5.15.0-113-generic-x86_64-with-glibc2.35
- Running on a notebook?: No
- Running on Google Colab?: No
- Python version: 3.10.14
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.24.0
- Transformers version: 4.41.2
- Accelerate version: 0.22.0
- PEFT version: 0.11.1
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: 0.0.23
- Accelerator: NVIDIA RTX A5000, 24564 MiB VRAM
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
### Who can help?
_No response_
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.