Controlnet inpainting change too much of masking area

### Describe the bug

mask



i got mask with this

```
def make_inpaint_condition (image, image_mask):
image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
image_mask = np.array(image_mask.convert("L")).astype(np.float32) / 255.0

assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
image[image_mask > 0.5] = -1.0 # set as masked pixel
image = np.expand_dims(image, 0).transpose(0, 3, 1, 2)
image = torch.from_numpy(image)
return image
```

if use apply mask still bad around edge

```python
output_image = pipe.image_processor.apply_overlay(mask_image1, input_image, output_image)
```


Because there are too many noise below, the one above cannot cover them.

is it possible to preserve mask area?

---

and below is when i use compel without DiffusersTextualInversionManager

the color changed, but it looks cloth is original

https://github.com/damian0815/compel/issues/86


### Reproduction

code from here

https://huggingface.co/docs/diffusers/en/using-diffusers/controlnet

### Logs

```shell
no log
```

### System Info

0.27.2

### Who can help?

_No response_