OSError: stabilityai/stable-diffusion-xl-base-1.0 does not appear to have a file named diffusion_pytorch_model.bin.

### Describe the bug

I use lora to fintune the model of 'stabilityai/stable-diffusion-xl-base-1.0'，get fellow error:
OSError: stabilityai/stable-diffusion-xl-base-1.0 does not appear to have a file named diffusion_pytorch_model.bin.

### Reproduction

python train_text_to_image_lora_sdxl.py --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" --dataset_name="/nobackup/sc23yl/MSC/lora/sd_xl_cartoon_new/cartoon_dataset" --caption_column="text" --resolution=1024 --random_flip --train_batch_size=1 --num_train_epochs=2 --checkpointing_steps=500 --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 --mixed_precision="fp16" --seed=42 --output_dir="/nobackup/sc23yl/MSC/lora/sd_xl_cartoon_new/fine-tune-lora" --validation_prompt="1girl, bangs, blush, crescent, crescent hair ornament, future parade (love live), grey hair, hair ornament, hat, looking at viewer, love live, love live nijigasaki high school idol club, nakasu kasumi, pink eyes, short hair, sidelocks, simple background, smile, star (symbol), star hair ornament, tomo wakui, upper body, white background"

### Logs

_No response_

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- 🤗 Diffusers version: 0.30.0.dev0
- Platform: Linux-3.10.0-1160.92.1.el7.x86_64-x86_64-with-glibc2.17
- Running on Google Colab?: No
- Python version: 3.12.3
- PyTorch version (GPU?): 2.3.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.23.4
- Transformers version: 4.43.3
- Accelerate version: 0.31.0
- PEFT version: 0.7.0
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: not installed
- Accelerator: NA
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help?

_No response_