Kohya started using more VRAM for SDXL and using more than it should be

I have a config which was running on Kaggle fine in previous versions

Right now it is failing on 15 GB gpu

This should not happen

Same settings on OneTrainer uses lesser than 13.5 GB VRAM

Here it fails with 15 GB

It wasn't failing before

All images are 1024x1024
All cached

Here the full training used prompt

I did trainings in past in Kaggle and this exact prompt was working i even have a video of it here

https://youtu.be/16-b1AjvyBE

```
accelerate launch --num_cpu_threads_per_process=4
"./sdxl_train.py" --max_grad_norm=0.0 --no_half_vae
--train_text_encoder --ddp_timeout=10000000
--ddp_gradient_as_bucket_view --bucket_no_upscale
--bucket_reso_steps=64 --cache_latents
--cache_latents_to_disk --full_fp16
--gradient_checkpointing --learning_rate="1e-05"
--learning_rate_te1="3e-06"
--logging_dir="/kaggle/working/results/log"
--lr_scheduler="constant" --lr_scheduler_num_cycles="1"
--max_data_loader_n_workers="0"
--resolution="1024,1024" --max_train_steps="1500"
--mem_eff_attn --mixed_precision="fp16"
--optimizer_args scale_parameter=False
relative_step=False warmup_init=False weight_decay=0.01
--optimizer_type="Adafactor"
--output_dir="/kaggle/working/results/model"
--output_name="2024_02_21_kaggle"
--pretrained_model_name_or_path="stabilityai/stable-dif
fusion-xl-base-1.0"
--reg_data_dir="/kaggle/working/results/reg"
--save_every_n_epochs="1" --save_model_as=safetensors
--save_precision="fp16" --train_batch_size="1"
--train_data_dir="/kaggle/working/results/img"
--vae="stabilityai/sdxl-vae" --xformers
```

```
Traceback (most recent call last):
File "/kaggle/working/kohya_ss/./sdxl_train.py", line 779, in
train(args)
File "/kaggle/working/kohya_ss/./sdxl_train.py", line 594, in train
optimizer.step()
File "/opt/conda/lib/python3.10/site-packages/accelerate/optimizer.py", line 132, in step
self.scaler.step(self.optimizer, closure)
File "/opt/conda/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 374, in step
retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 290, in _maybe_opt_step
retval = optimizer.step(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/accelerate/optimizer.py", line 185, in patched_step
return method(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 69, in wrapper
return wrapped(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/optim/optimizer.py", line 280, in wrapper
out = func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/optimization.py", line 715, in step
update = (grad**2) + group["eps"][0]
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 242.00 MiB (GPU 1; 14.75 GiB total capacity; 14.34 GiB already allocated; 53.06 MiB free; 14.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

Start by reproducing the supplied accelerate command with the cached 1024x1024 dataset and inspect the optimizer.step() path around line 594 of sdxl_train.py. Compare memory use with the earlier version referenced in the report and determine why the same settings now exceed the 15 GB GPU limit. Done means the configuration completes training without the reported CUDA out-of-memory failure, or the regression is clearly narrowed down.

Written by the indexing model from the issue text.