Issue with reproducing ip_adapter_sdxl_controlnet_demo.ipynb example from doc

Hi all!

I met an issue when I tried to use diffusers/hugging face syntax to reproduce an [example](https://github.com/tencent-ailab/IP-Adapter/blob/main/ip_adapter_sdxl_controlnet_demo.ipynb).

I am obtaining different/worse result and wondering why...


Here my [gist](https://colab.research.google.com/gist/katarzynasornat/ff9bc215c39ab6d62e5d54b55305a591/ip_adapter_example.ipynb)

Any gist or advice very welcome!

Thank you for your work and effort.