'UNet2DConditionModel' object has no attribute 'peft_config'

### Describe the bug

After pulling the latest code from the _examples/research_projects/instructpix2pix_lora_ repository, I encountered the following error when running the default code:
`AttributeError: 'Linear' object has no attribute 'set_lora_layer'`
I resolved this issue by modifying the code.
The training now runs successfully, but when I try to save the model, I encounter another error:
`Traceback (most recent call last):7%|██████████████████████████████████████████████████████████████████████████████████████▊ | 4/7 [00:00<00:00, 17.90it/s]
File "/home/guying/diffusers/examples/research_projects/instructpix2pix_lora/train_instruct_pix2pix_lora.py", line 1160, in
main()
File "/home/guying/diffusers/examples/research_projects/instructpix2pix_lora/train_instruct_pix2pix_lora.py", line 1120, in main
unet.save_attn_procs(args.output_dir, weights_name="pytorch_lora_weights.bin")
File "/home/guying/diffusers/src/diffusers/loaders/unet.py", line 500, in save_attn_procs
state_dict = get_peft_model_state_dict(self)
File "/home/guying/anaconda3/envs/diffuser/lib/python3.9/site-packages/peft/utils/save_and_load.py", line 73, in get_peft_model_state_dict
config = model.peft_config[adapter_name]
File "/home/guying/diffusers/src/diffusers/models/modeling_utils.py", line 155, in __getattr__
return super().__getattr__(name)
File "/home/guying/anaconda3/envs/diffuser/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1931, in __getattr__
raise AttributeError(
AttributeError: 'UNet2DConditionModel' object has no attribute 'peft_config'`

### Reproduction

Pull the latest code from the repository.
Apply the code modification to train the model.
`
# # referred to https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py
# unet_lora_parameters = []
# for attn_processor_name, attn_processor in unet.attn_processors.items():
# # Parse the attention module.
# attn_module = unet
# for n in attn_processor_name.split(".")[:-1]:
# attn_module = getattr(attn_module, n)
# print(type(attn_module.to_q)) # 添加调试输出

# # Set the `lora_layer` attribute of the attention-related matrices.
# attn_module.to_q.set_lora_layer(
# LoRALinearLayer(
# in_features=attn_module.to_q.in_features, out_features=attn_module.to_q.out_features, rank=args.rank
# )
# )
# attn_module.to_k.set_lora_layer(
# LoRALinearLayer(
# in_features=attn_module.to_k.in_features, out_features=attn_module.to_k.out_features, rank=args.rank
# )
# )

# attn_module.to_v.set_lora_layer(
# LoRALinearLayer(
# in_features=attn_module.to_v.in_features, out_features=attn_module.to_v.out_features, rank=args.rank
# )
# )
# attn_module.to_out[0].set_lora_layer(
# LoRALinearLayer(
# in_features=attn_module.to_out[0].in_features,
# out_features=attn_module.to_out[0].out_features,
# rank=args.rank,
# )
# )

# # Accumulate the LoRA params to optimize.
# unet_lora_parameters.extend(attn_module.to_q.lora_layer.parameters())
# unet_lora_parameters.extend(attn_module.to_k.lora_layer.parameters())
# unet_lora_parameters.extend(attn_module.to_v.lora_layer.parameters())
# unet_lora_parameters.extend(attn_module.to_out[0].lora_layer.parameters())

unet_lora_parameters = []
for attn_processor_name, attn_processor in unet.attn_processors.items():
# Parse the attention module.
attn_module = unet
for n in attn_processor_name.split(".")[:-1]:
attn_module = getattr(attn_module, n)

# Replace the Linear layers with LoRALinearLayer
# Creating new LoRA layers directly replacing the existing layers
setattr(attn_module, 'to_q', LoRALinearLayer(
in_features=attn_module.to_q.in_features,
out_features=attn_module.to_q.out_features,
rank=args.rank)
)
setattr(attn_module, 'to_k', LoRALinearLayer(
in_features=attn_module.to_k.in_features,
out_features=attn_module.to_k.out_features,
rank=args.rank)
)
setattr(attn_module, 'to_v', LoRALinearLayer(
in_features=attn_module.to_v.in_features,
out_features=attn_module.to_v.out_features,
rank=args.rank)
)
# Assuming `to_out` is a sequence and the first element is the linear layer
setattr(attn_module.to_out, '0', LoRALinearLayer(
in_features=attn_module.to_out[0].in_features,
out_features=attn_module.to_out[0].out_features,
rank=args.rank)
)

# Add the parameters of the newly created LoRA layers to the parameters list
unet_lora_parameters.extend(attn_module.to_q.parameters())
unet_lora_parameters.extend(attn_module.to_k.parameters())
unet_lora_parameters.extend(attn_module.to_v.parameters())
unet_lora_parameters.extend(attn_module.to_out[0].parameters())
`

I tried the following bash command to launch the training:
`export MODEL_ID="timbrooks/instruct-pix2pix"
export DATASET_ID="keaiguogongzhu/relight-dataset"
export OUTPUT_DIR="/home/guying/relight_nerf/instruct-nerf2nerf/instructPix2Pix-relighting_4"

accelerate launch /home/guying/diffusers/examples/research_projects/instructpix2pix_lora/train_instruct_pix2pix_lora.py \
--pretrained_model_name_or_path=$MODEL_ID \
--dataset_name=$DATASET_ID \
--enable_xformers_memory_efficient_attention \
--resolution=256 --random_flip \
--train_batch_size=4 --gradient_accumulation_steps=2 --gradient_checkpointing \
--max_train_steps=100 \
--checkpointing_steps=50 --checkpoints_total_limit=5 \
--learning_rate=5e-05 --lr_warmup_steps=0 \
--seed=42 \
--rank=4 \
--output_dir=$OUTPUT_DIR \
--report_to=wandb `

### Logs

_No response_

### System Info

- 🤗 Diffusers version: 0.32.0.dev0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.31
- Running on Google Colab?: No
- Python version: 3.9.20
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.26.2
- Transformers version: 4.46.3
- Accelerate version: 1.1.1
- PEFT version: 0.13.2
- Bitsandbytes version: 0.44.1
- Safetensors version: 0.4.5
- xFormers version: 0.0.28.post3
- Accelerator: NVIDIA GeForce RTX 3080 Ti Laptop GPU, 16384 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

@sayakpaul Hi, sorry to bother you, but I’ve run into a problem with my modifications and am hoping you could assist me in resolving it. I really appreciate any help you can provide!