SD 1.5 DreamBooth training uses more VRAM than SDXL DreamBooth

I am using same settings in both cases and how much VRAM SD 1.5 uses is insane

Here below the full config.

All training and reg images are 768x768

However SD 1.5 training faster than SDXL. That is the only expected thing :)

SDXL uses 17 GB VRAM meanwhile SD 1.5 uses 22.5 GB - tested on RunPod Linux - no desktop GUI

```
accelerate launch --num_cpu_threads_per_process=4 "./train_db.py" --pretrained_model_name_or_path="/workspace/stable-diffusion-webui/models/Stable-diffusion/hyper_real_v3.safetensors" --train_data_dir="/workspace/train" --reg_data_dir="/workspace/reg"
--resolution="768,768" --output_dir="/workspace/stable-diffusion-webui/models/Stable-diffusion" --logging_dir="/workspace/stable-diffusion-webui/models/Stable-diffusion" --save_model_as=safetensors --full_bf16 --output_name="6e5"
--lr_scheduler_num_cycles="1" --max_data_loader_n_workers="0" --learning_rate_te="6e-05" --learning_rate="6e-05" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="4500" --save_every_n_epochs="1" --mixed_precision="bf16"
--save_precision="bf16" --cache_latents --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False weight_decay=0.01 --max_data_loader_n_workers="0" --bucket_reso_steps=64
--gradient_checkpointing --bucket_no_upscale --noise_offset=0.0 --max_grad_norm=0.0 --no_half_vae
```

Start with the train_db.py entry point and reproduce the supplied accelerate launch configuration on RunPod Linux, comparing the SD 1.5 and SDXL DreamBooth runs at 768x768. Trace the training configuration and memory usage to identify the cause of the reported VRAM difference, then document or correct the behavior and verify both runs.

Written by the indexing model from the issue text.