pipeline_stable_diffusion_xl_adapter

### Describe the bug

I want to rewrite the call function of the pipeline_stable_diffusion_xl_adapter. When I want to use the function prepare_ip_adapter_image_embeds, there is an error called "AttributeError: 'NoneType' object has no attribute 'image_projection_layers'". The error tells me that the attribution self.unet.encoder_hid_proj is 'NoneType'. The pre-trianed model is 'stabilityai/stable-diffusion-xl-base-1.0'. Is there anything wrong when I use it? Thank you.

### Reproduction

model_path = 'stabilityai/stable-diffusion-xl-base-1.0'
adapter = T2IAdapter.from_pretrained("TencentARC/t2i-adapter-openpose-sdxl-1.0",)
scheduler = DDPMScheduler.from_pretrained(model_path, subfolder="scheduler")
pipe = AdapterPosePipeline.from_pretrained(model_path, adapter=adapter, torch_dtype=torch.float16, variant="fp16", scheduler=scheduler).to(device)

image_embeds = self.prepare_ip_adapter_image_embeds(
image,
ip_adapter_image_embeds,
device,
batch_size * num_images_per_prompt,
self.do_classifier_free_guidance,
)

### Logs

```shell
root@autodl-container-9d8d46936f-161f523c:~/autodl-tmp/COMP5704_Pose_Driven/src# python run.py
/root/miniconda3/lib/python3.12/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
/root/miniconda3/lib/python3.12/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
warnings.warn(
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00, 4.87it/s]
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/open_pose/body.py:34: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model_dict = util.transfer(self.model, torch.load(model_path))
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/open_pose/hand.py:14: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model_dict = util.transfer(self.model, torch.load(model_path))
/root/miniconda3/lib/python3.12/site-packages/controlnet_aux/open_pose/face.py:325: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
self.model.load_state_dict(torch.load(face_model_path))
[(0.44140625, 0.125), (0.462890625, 0.23046875), (0.37890625, 0.240234375), (0.337890625, 0.353515625), (0.2734375, 0.46484375), (0.548828125, 0.220703125), (0.591796875, 0.353515625), (0.6015625, 0.47265625), (0.380859375, 0.462890625), (0.33203125, 0.6640625), (0.341796875, 0.900390625), (0.5078125, 0.458984375), (0.53125, 0.66015625), (0.640625, 0.8203125), (0.416015625, 0.109375), (0.458984375, 0.10546875), (0.390625, 0.1328125), (0.494140625, 0.12109375)]
Traceback (most recent call last):
File "/root/autodl-tmp/COMP5704_Pose_Driven/src/run.py", line 94, in
result = pipe(latent_list, "", "", pose_map, image, clip_image, target_coord, src_coord, num_inference_steps=50).images[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/COMP5704_Pose_Driven/src/pose_pipeline_adapter.py", line 550, in __call__
image_embeds = self.prepare_ip_adapter_image_embeds(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py", line 557, in prepare_ip_adapter_image_embeds
if len(ip_adapter_image) != len(self.unet.encoder_hid_proj.image_projection_layers):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'image_projection_layers'
```

### System Info

absl-py==2.1.0
accelerate==0.33.0
addict==2.4.0
ai_tools==0.3.9
aiohappyeyeballs==2.3.5
aiohttp==3.10.3
aiosignal==1.3.1
anaconda-anon-usage @ file:///croot/anaconda-anon-usage_1710965072196/work
antlr4-python3-runtime==4.9.3
anyio==4.4.0
archspec @ file:///croot/archspec_1709217642129/work
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
attrs==23.2.0
Babel==2.15.0
basicsr==1.4.2
beautifulsoup4==4.12.3
bleach==6.1.0
boltons @ file:///work/perseverance-python-buildout/croot/boltons_1698851177130/work
Brotli @ file:///croot/brotli-split_1714483155106/work
certifi @ file:///croot/certifi_1707229174982/work/certifi
cffi @ file:///croot/cffi_1714483155441/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
cognitive-face==1.5.0
comm==0.2.2
conda @ file:///croot/conda_1714403036266/work
conda-content-trust @ file:///croot/conda-content-trust_1714483159009/work
conda-libmamba-solver @ file:///croot/conda-libmamba-solver_1706733287605/work/src
conda-package-handling @ file:///croot/conda-package-handling_1714483155348/work
conda_package_streaming @ file:///work/perseverance-python-buildout/croot/conda-package-streaming_1698847176583/work
contourpy==1.2.1
controlnet-aux==0.0.9
cryptography @ file:///croot/cryptography_1714660666131/work
cycler==0.12.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
diffusers==0.29.2
distro @ file:///croot/distro_1714488253808/work
dlib==19.24.6
einops==0.8.0
executing==2.0.1
fastjsonschema==2.19.1
filelock==3.14.0
fonttools==4.53.0
fqdn==1.5.1
frozenlist==1.4.1
fsspec==2024.5.0
future==1.0.0
grpcio==1.64.0
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
huggingface-hub==0.24.5
idna @ file:///croot/idna_1714398848350/work
imageio==2.34.2
importlib_metadata==8.2.0
ipykernel==6.29.4
ipython==8.25.0
ipywidgets==8.1.3
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpatch @ file:///croot/jsonpatch_1714483231291/work
jsonpointer==2.1
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyter_server==2.14.1
jupyter_server_terminals==0.5.3
jupyterlab==4.2.1
jupyterlab-language-pack-zh-CN==4.2.post1
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.2
jupyterlab_widgets==3.0.11
kiwisolver==1.4.5
lazy_loader==0.4
ldm==0.1.3
libmambapy @ file:///croot/mamba-split_1714483352891/work/libmambapy
lightning-utilities==0.11.6
lmdb==1.5.1
lpips==0.1.4
Markdown==3.6
MarkupSafe==2.1.5
matplotlib==3.9.0
matplotlib-inline==0.1.7
menuinst @ file:///croot/menuinst_1714510563922/work
mistune==3.0.2
mpmath==1.3.0
multidict==6.0.5
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
notebook_shim==0.2.4
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.40
nvidia-nvtx-cu12==12.1.105
omegaconf==2.3.0
opencv-python==4.10.0.84
opencv-python-headless==4.10.0.84
overrides==7.7.0
packaging @ file:///croot/packaging_1710807400464/work
pandas==2.2.2
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
pillow==10.3.0
platformdirs @ file:///work/perseverance-python-buildout/croot/platformdirs_1701732573265/work
pluggy @ file:///work/perseverance-python-buildout/croot/pluggy_1698805497733/work
prometheus_client==0.20.0
prompt_toolkit==3.0.45
protobuf==5.27.0
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pycosat @ file:///croot/pycosat_1714510623388/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments==2.18.0
pyparsing==3.1.2
PySocks @ file:///work/perseverance-python-buildout/croot/pysocks_1698845478203/work
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytorch-lightning==2.4.0
pytz==2024.1
PyYAML==6.0.1
pyzmq==26.0.3
referencing==0.35.1
regex==2024.7.24
requests @ file:///croot/requests_1707355572290/work
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.18.1
ruamel.yaml @ file:///work/perseverance-python-buildout/croot/ruamel.yaml_1698863605521/work
safetensors==0.4.3
scikit-image==0.24.0
scipy==1.14.0
seaborn==0.13.2
Send2Trash==1.8.3
setuptools==69.5.1
six==1.16.0
sniffio==1.3.1
soupsieve==2.5
stack-data==0.6.3
supervisor==4.2.5
sympy==1.12.1
tb-nightly==2.18.0a20240814
tensorboard==2.16.2
tensorboard-data-server==0.7.2
terminado==0.18.1
tifffile==2024.7.24
timm==0.6.7
tinycss2==1.3.0
tokenizers==0.19.1
tomli==2.0.1
torch==2.4.0
torchaudio==2.4.0
torchmetrics==1.4.1
torchvision==0.19.0
tornado==6.4
tqdm @ file:///croot/tqdm_1714567712644/work
traitlets==5.14.3
transformers==4.43.0
triton==3.0.0
truststore @ file:///work/perseverance-python-buildout/croot/truststore_1701735771625/work
types-python-dateutil==2.9.0.20240316
typing_extensions==4.12.1
tzdata==2024.1
uri-template==1.3.0
urllib3 @ file:///croot/urllib3_1707770551213/work
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.8.0
Werkzeug==3.0.3
wheel==0.43.0
widgetsnbextension==4.0.11
xformers==0.0.27.post2
yapf==0.40.2
yarl==1.9.4
zipp==3.19.2
zprint==0.0.11

### Who can help?

@sayakpaul @DN6 @yiyi

pose_pipeline_adapter.py の __call__ パスと、インストール済みの pipeline_stable_diffusion_xl_adapter.py にある prepare_ip_adapter_image_embeds の実装から始めます。stabilityai/stable-diffusion-xl-base-1.0 を使って、提示された AdapterPosePipeline のセットアップを再現し、その後、呼び出しが self.unet.encoder_hid_proj に到達したときの想定される動作を確認します。報告された AttributeError が解消されるか、互換性のないセットアップが明確に特定されれば完了です。

索引モデルが issue の本文から書いたものです。