DreamBooth LoRA SDXL OOM issue at the last inference (8xRTX3090 24G)

Hi, I'm encountering a CUDA out of memory error during the final inference step when running the 3D Icon example from the [advanced diffusion training](https://github.com/huggingface/diffusers/tree/main/examples/advanced_diffusion_training#3d-icon-example).

Script: I used the exact same script provided in the example.
Issue: Training epochs complete successfully without any OOM errors.
But the error occurs during the last inference step at [L2416](https://github.com/huggingface/diffusers/blob/6fe05b9b93593bca41afac79b32b7a23526b0e96/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py#L2416).

Tried Solutions:
Reduced --num_validation_images to 1, but the OOM still happens.

Environment:
GPUs: 8 x RTX3090 24GB
PyTorch version: 2.6.0
CUDA version: 12.4
diffusers version: 0.33.0.dev0