High Committed Memory Usage During Model Training with SDXL

I am currently training a large model(Checkpoint) using SDXL with a dataset of 13,820 images. I've observed a phenomenon where the program requests a significant amount of memory, but the actual usage is relatively low.

After enabling caching for latent variables and other parameters, the Committed Memory in Windows Task Manager shows up to 60GB, whereas the actual memory usage is around 10GB. This does not cause any issues during normal training, but problems arise when the training completes an epoch and attempts to output a model.

At that time, the upper limit of my Committed Memory was 63.8GB. Since the memory usage was close to this limit (60GB), the system failed to export the model and displayed an error indicating insufficient memory. The issue persisted until I reduced the dataset to 300 images and disabled latent variable caching, which then allowed the model to be output successfully.

To resolve this, I increased the virtual memory to 91.5GB. I am wondering if this behavior is expected, or if there might be a more efficient way to manage memory allocation to prevent such high Committed Memory usage.


No files, tests, or entry points are named in the report. Start by reproducing SDXL training with 13,820 images and latent caching on Windows while monitoring committed versus actual memory. Done means determining whether the allocation is expected and whether model export can avoid failing near the commit limit.

Written by the indexing model from the issue text.