Issue with reproducing ip_adapter_sdxl_controlnet_demo.ipynb example from doc

Hi all!

I met an issue when I tried to use diffusers/hugging face syntax to reproduce an [example](https://github.com/tencent-ailab/IP-Adapter/blob/main/ip_adapter_sdxl_controlnet_demo.ipynb).

I am obtaining different/worse result and wondering why...


Here my [gist](https://colab.research.google.com/gist/katarzynasornat/ff9bc215c39ab6d62e5d54b55305a591/ip_adapter_example.ipynb)

Any gist or advice very welcome!

Thank you for your work and effort.

Start by comparing the linked ip_adapter_sdxl_controlnet_demo.ipynb example with the reporter's Colab gist, focusing on the diffusers/Hugging Face syntax and the differing output shown. Reproduce the result if possible and document the cause of the discrepancy or the changes needed for the example to match.

索引モデルが issue の本文から書いたものです。