Many time passes between epochs

Nobody has claimed this yet.

### Bug description

I am training the model using pytorch-lightning.
The code is below

One epoch passes smoothly in 1-2 seconds, however the interval between epochs is around 6-8 seconds. Could you recommed any guidelines where to search and how to fix it?

```
train_dataloader = torch.utils.data.DataLoader(
train_dataset,
batch_size=config.training.batch_size,
shuffle=True,
num_workers=2,
#pin_memory=True,
persistent_workers=True
)

model = instantiate(config.model_cfg.model_obj)
pl_model = PlWaveGrad(model, hydra_config=config)

if config.model_cfg.conditional:
print('using conditional model')
else:
print('uncondtional model')

print('using dataset: ', config.dataset.name)


loss_cb = LossCurveCallback()

#profiler = AdvancedProfiler(dirparth='./outputs', filename='perf_logs')

checkpoint_callback = pl.callbacks.ModelCheckpoint(
filename="{epoch:02d}-{train_loss:.4f}",
every_n_epochs=1000, # Save after every epoch
save_last=True, # This ensures the last checkpoint is kept and overwritten each time
)
trainer = pl.Trainer(
max_epochs=config.training.n_epochs,
log_every_n_steps=4,
devices=[config.training.device], # FIXME
callbacks=[checkpoint_callback, loss_cb], # here is no loss_cb
precision='16-mixed',
#enable_checkpointing=False,
#profiler="advanced",
)

print(f'\n\n\n using device: {config.training.device}')

if config.ckpt_path is None:
trainer.fit(pl_model, train_dataloaders=train_dataloader)

else:
ckpt_path = Path(config.ckpt_path).resolve()
trainer.fit(pl_model, train_dataloaders=train_dataloader, ckpt_path=ckpt_path)

```

### What version are you seeing the problem on?

v2.2, v2.4

### Reproduced in studio

_No response_

### How to reproduce the bug

```python

```

### Error messages and logs

```
# Error messages and logs here please
```

### Environment

Current environment

```
#- PyTorch Lightning Version (e.g., 2.6.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
```

### More info

_No response_

cc @ethanwharris

Start by reproducing the reported training loop with the supplied DataLoader and Trainer configuration, after filling in the missing Lightning, PyTorch, Python, OS, CUDA, and GPU versions. Use Lightning's profiler or the commented AdvancedProfiler configuration to identify whether the delay occurs in data loading, callbacks, checkpointing, or epoch hooks. Done means a minimal reproduction, a measured cause, and a verified fix or documented limitation.

Written by the indexing model from the issue text.