StableDiffusionXLControlNetImg2ImgPipeline often fails to respect "pose" control images

### Describe the bug
Hello,

Using [StableDiffusionXLControlNetImg2ImgPipeline](https://huggingface.co/docs/diffusers/en/api/pipelines/controlnet_sdxl#diffusers.StableDiffusionXLControlNetImg2ImgPipeline), and passing a "pose" control image often fails to produce an output image that maintains the pose.
I couldn't find much info about this pipeline used with a pose image; I'd like to know whether the problem comes from the underlying pipe not being able to run an inference with this conditioning or if I'm doing something wrong, eg haven't found the right params.

Note that on the link above the example snippet uses a canny image; and that the [controlnet model I'm using](https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0) uses a pose image but the `StableDiffusionXLControlNetPipeline` pipeline instead of `StableDiffusionXLControlNetImg2ImgPipeline`.

In the snippet, belows, [the control image used](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/person.png) works, however most control images will fail in my expeirence.

How to get this pipeline to consistently respect the pose from `control_image` ?

Thanks,

### Reproduction

Using this image as reference image for the img2img part:

```python
import torch

from controlnet_aux import OpenposeDetector
from diffusers import ControlNetModel
from diffusers import StableDiffusionXLControlNetImg2ImgPipeline
from diffusers.utils import load_image
from PIL import Image

controlnet = ControlNetModel.from_pretrained(
"thibaud/controlnet-openpose-sdxl-1.0",
torch_dtype=torch.float16,
)
pipe = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
controlnet=controlnet,
variant="fp16",
use_safetensors=True,
torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()
openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
pose_image = load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/person.png",
)
control_image = openpose(pose_image).resize((1024, 1024))
control_image.save('control.png')

prompt = "daiton style, daiton, A brave sheriff with a star badge, wearing a cowboy hat and spurs, standing confidently, illustration style, minimalist, illustration style, minimalist, solid color background"
negative_prompt = "blurry, ugly, messy weird"
image = Image.open(
,
).resize((1024, 1024))
controlnet_conditioning_scale = 1.0

images = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
image=image,
control_image=control_image,
strength=1.0,
num_inference_steps=30,
controlnet_conditioning_scale=controlnet_conditioning_scale,
).images
images[0].save("from_diffusers.png")
```

Things I have tried:
- various params (eg guidance scale, more steps)
- other pose image
- thicker edges in the pose image
- other image sizes

### Logs

```shell
/home/ubuntu/anaconda3/envs/inference_v2/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
Loading pipeline components...: 100%|█| 7/7 [00:00<
100%|██████████████| 30/30 [00:14<00:00, 2.12it/s]
```

### System Info

- `diffusers` version: 0.27.2
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- PyTorch version (GPU?): 2.1.0 (True)
- Huggingface_hub version: 0.23.1
- Transformers version: 4.39.3
- Accelerate version: 0.25.0
- xFormers version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help?

@yiyixuxu @sayakpaul @DN6