pipeline_stable_diffusion_xl_adapter

### Describe the bug

I want to rewrite the call function of the pipeline_stable_diffusion_xl_adapter. When I want to use the function prepare_ip_adapter_image_embeds, there is an error called "AttributeError: 'NoneType' object has no attribute 'image_projection_layers'". The error tells me that the attribution self.unet.encoder_hid_proj is 'NoneType'. The pre-trianed model is 'stabilityai/stable-diffusion-xl-base-1.0'. Is there anything wrong when I use it? Thank you.

### Reproduction

model_path = 'stabilityai/stable-diffusion-xl-base-1.0'
adapter = T2IAdapter.from_pretrained("TencentARC/t2i-adapter-openpose-sdxl-1.0",)
scheduler = DDPMScheduler.from_pretrained(model_path, subfolder="scheduler")
pipe = AdapterPosePipeline.from_pretrained(model_path, adapter=adapter, torch_dtype=torch.float16, variant="fp16", scheduler=scheduler).to(device)

image_embeds = self.prepare_ip_adapter_image_embeds(
image,
ip_adapter_image_embeds,
device,
batch_size * num_images_per_prompt,
self.do_classifier_free_guidance,
)

### Logs

```shell
root@autodl-container-9d8d46936f-161f523c:~/autodl-tmp/COMP5704_Pose_Driven/src# python run.py
/root/miniconda3/lib/python3.12/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
/root/miniconda3/lib/python3.12/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
warnings.warn(
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00, 4.87it/s]
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/open_pose/body.py:34: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model_dict = util.transfer(self.model, torch.load(model_path))
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/open_pose/hand.py:14: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model_dict = util.transfer(self.model, torch.load(model_path))
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/open_pose/face.py:325: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
self.model.load_state_dict(torch.load(face_model_path))
[(0.44140625, 0.125), (0.462890625, 0.23046875), (0.37890625, 0.240234375), (0.337890625, 0.353515625), (0.2734375, 0.46484375), (0.548828125, 0.220703125), (0.591796875, 0.353515625), (0.6015625, 0.47265625), (0.380859375, 0.462890625), (0.33203125, 0.6640625), (0.341796875, 0.900390625), (0.5078125, 0.458984375), (0.53125, 0.66015625), (0.640625, 0.8203125), (0.416015625, 0.109375), (0.458984375, 0.10546875), (0.390625, 0.1328125), (0.494140625, 0.12109375)]
Traceback (most recent call last):
File "/root/autodl-tmp/COMP5704_Pose_Driven/src/run.py", line 94, in
result = pipe(latent_list, "", "", pose_map, image, clip_image, target_coord, src_coord, num_inference_steps=50).images[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/COMP5704_Pose_Driven/src/pose_pipeline_adapter.py", line 550, in __call__
image_embeds = self.prepare_ip_adapter_image_embeds(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py", line 557, in prepare_ip_adapter_image_embeds
if len(ip_adapter_image) != len(self.unet.encoder_hid_proj.image_projection_layers):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'image_projection_layers'
```

### System Info

absl-py==2.1.0
accelerate==0.33.0
addict==2.4.0
ai_tools==0.3.9
aiohappyeyeballs==2.3.5
aiohttp==3.10.3
aiosignal==1.3.1
anaconda-anon-usage @ file:///croot/anaconda-anon-usage_1710965072196/work
antlr4-python3-runtime==4.9.3
anyio==4.4.0
archspec @ file:///croot/archspec_1709217642129/work
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
attrs==23.2.0
Babel==2.15.0
basicsr==1.4.2
beautifulsoup4==4.12.3
bleach==6.1.0
boltons @ file:///work/perseverance-python-buildout/croot/boltons_1698851177130/work
Brotli @ file:///croot/brotli-split_1714483155106/work
certifi @ file:///croot/certifi_1707229174982/work/certifi
cffi @ file:///croot/cffi_1714483155441/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
cognitive-face==1.5.0
comm==0.2.2
conda @ file:///croot/conda_1714403036266/work
conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work
conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1706733287605/work/src
conda-package-handling @ file:///croot/conda-package-handling_1714483155348/work
conda_package_streaming @ file:///work/perseverance-python-buildout/croot/conda-package-streaming_1698847176583/work
contourpy==1.2.1
controlnet-aux==0.0.9
cryptography @ file:///croot/cryptography_1714660666131/work
cycler==0.12.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
diffusers==0.29.2
distro @ file:///croot/distro_1714488253808/work
dlib==19.24.6
einops==0.8.0
executing==2.0.1
fastjsonschema==2.19.1
filelock==3.14.0
fonttools==4.53.0
fqdn==1.5.1
frozenlist==1.4.1
fsspec==2024.5.0
future==1.0.0
grpcio==1.64.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
huggingface-hub==0.24.5
idna @ file:///croot/idna_1714398848350/work
imageio==2.34.2
importlib_metadata==8.2.0
ipykernel==6.29.4
ipython==8.25.0
ipywidgets==8.1.3
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpatch @ file:///croot/jsonpatch_1714483231291/work
jsonpointer==2.1
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyter_server==2.14.1
jupyter_server_terminals==0.5.3
jupyterlab==4.2.1
jupyterlab-language-pack-zh-CN==4.2.post1
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.2
jupyterlab_widgets==3.0.11
kiwisolver==1.4.5
lazy_loader==0.4
ldm==0.1.3
libmambapy @ file:///croot/mamba-split_1714483352891/work/libmambapy
lightning-utilities==0.11.6
lmdb==1.5.1
lpips==0.1.4
Markdown==3.6
MarkupSafe==2.1.5
matplotlib==3.9.0
matplotlib-inline==0.1.7
menuinst @ file:///croot/menuinst_1714510563922/work
mistune==3.0.2
mpmath==1.3.0
multidict==6.0.5
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
notebook_shim==0.2.4
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.40
nvidia-nvtx-cu12==12.1.105
omegaconf==2.3.0
opencv-python==4.10.0.84
opencv-python-headless==4.10.0.84
overrides==7.7.0
packaging @ file:///croot/packaging_1710807400464/work
pandas==2.2.2
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
pillow==10.3.0
platformdirs @ file:///work/perseverance-python-buildout/croot/platformdirs_1701732573265/work
pluggy @ file:///work/perseverance-python-buildout/croot/pluggy_1698805497733/work
prometheus_client==0.20.0
prompt_toolkit==3.0.45
protobuf==5.27.0
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pycosat @ file:///croot/pycosat_1714510623388/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments==2.18.0
pyparsing==3.1.2
PySocks @ file:///work/perseverance-python-buildout/croot/pysocks_1698845478203/work
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytorch-lightning==2.4.0
pytz==2024.1
PyYAML==6.0.1
pyzmq==26.0.3
referencing==0.35.1
regex==2024.7.24
requests @ file:///croot/requests_1707355572290/work
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.18.1
ruamel.yaml @ file:///work/perseverance-python-buildout/croot/ruamel.yaml_1698863605521/work
safetensors==0.4.3
scikit-image==0.24.0
scipy==1.14.0
seaborn==0.13.2
Send2Trash==1.8.3
setuptools==69.5.1
six==1.16.0
sniffio==1.3.1
soupsieve==2.5
stack-data==0.6.3
supervisor==4.2.5
sympy==1.12.1
tb-nightly==2.18.0a20240814
tensorboard==2.16.2
tensorboard-data-server==0.7.2
terminado==0.18.1
tifffile==2024.7.24
timm==0.6.7
tinycss2==1.3.0
tokenizers==0.19.1
tomli==2.0.1
torch==2.4.0
torchaudio==2.4.0
torchmetrics==1.4.1
torchvision==0.19.0
tornado==6.4
tqdm @ file:///croot/tqdm_1714567712644/work
traitlets==5.14.3
transformers==4.43.0
triton==3.0.0
truststore @ file:///work/perseverance-python-buildout/croot/truststore_1701735771625/work
types-python-dateutil==2.9.0.20240316
typing_extensions==4.12.1
tzdata==2024.1
uri-template==1.3.0
urllib3 @ file:///croot/urllib3_1707770551213/work
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.8.0
Werkzeug==3.0.3
wheel==0.43.0
widgetsnbextension==4.0.11
xformers==0.0.27.post2
yapf==0.40.2
yarl==1.9.4
zipp==3.19.2
zprint==0.0.11

### Who can help?

@sayakpaul @DN6 @yiyi