FluxFillPipeline quality deteriorates over multiple iterations of inpainting

### Describe the bug

When the FluxFillPipeline is run multiple times on the same image, the image quality deteriorates in each iteration.

### Reproduction

```python
import torch
from diffusers import FluxFillPipeline
from diffusers.utils import load_image

pipe = FluxFillPipeline.from_pretrained("black-forest-labs/FLUX.1-Fill-dev", torch_dtype=torch.bfloat16)
pipe = pipe.to("cuda")

img = load_image("https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/cup.png")
mask = load_image("https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/cup_mask.png")

def fill(prompt, image, mask):
return pipe(
prompt=prompt,
image=image,
mask_image=mask,
guidance_scale=30,
num_inference_steps=50,
max_sequence_length=512,
height=1632,
width=1232,
generator=torch.Generator("cpu").manual_seed(0),
).images[0]

prompt = "a white paper cup"
for i in range(12):
print(i)
img = fill(prompt, img, mask)
img.save(f"output_{i}.png")
```

### Logs


### System Info

Diffusers Version 0.31.0
Python Version 3.10
CUDA Version 12.1.0
DGX H100

### Who can help?

@yiyixuxu @DN6 @aso