LORA caption training: extremely long pauses between epochs

For some reason there is a large delay when epochs change, making training much slower, what could cause this?
My settings:
accelerate launch --num_cpu_threads_per_process 10 train_network.py --pretrained_model_name_or_path=B:\AIimages\stable-diffusion-webui\models\Stable-diffusion\model.ckpt --train_data_dir=B:\AIimages\training\data --output_dir=B:\train\out\ --in_json=B:\AIimages\training\data\meta_lat.json --resolution=512,512 --prior_loss_weight=1.0 --train_batch_size=4 --learning_rate=1e-3 --max_train_steps=15000 --use_8bit_adam --xformers --gradient_checkpointing --mixed_precision=fp16 --save_every_n_epochs=10 --network_module=networks.lora --shuffle_caption --unet_lr=3e-4 --text_encoder_lr=3e-5 --lr_scheduler=constant --save_model_as=safetensors --seed=115

Start with train_network.py and reproduce the epoch transition using the command and settings in the issue, timing the pauses between epochs. Trace what runs at the transition and compare it with normal training steps. Done means identifying the cause of the delay and verifying that the resulting change removes or materially reduces it.