sdxl lora training seems to consume much more gpu memory than normal sdxl training does?

Specifically, sdxl_train v.s. sdxl_train_network
I have compared the trainable params, the are the same, and the training params are the same.
As a result, batch size 10 --> 4 otherwise an gpu memory error will occur.
So a lora module[64dim 32 alpha] consumes 6 batch size.
Any one give me some hints, thanks!

Start by comparing the sdxl_train and sdxl_train_network entry points with the reported matching trainable and training parameters. Reproduce the GPU-memory difference using the stated batch-size change and 64-dimension, 32-alpha LoRA settings; done means identifying the cause and preventing the unexpected memory increase or documenting the required memory behavior.

Written by the indexing model from the issue text.