Memory blows up when training large models on all TPU cores

Nobody has claimed this yet.

## 🐛 Bug

I am training on 8 TPU cores but the memory blows up when the epoch ends.

### To Reproduce

Try training a bert large on 8 TPU cores

### Expected behavior

Second epoch should get started

### Environment

Kaggle TPU

- PyTorch Lightning Version (e.g., 1.5.0):
- PyTorch Version (e.g., 1.10):
- Python version (e.g., 3.9):
- OS (e.g., Linux): Linux
- CUDA/cuDNN version:
- GPU models and configuration:
- How you installed PyTorch (`conda`, `pip`, source):
- If compiling from source, the output of `torch.__config__.show()`:
- Any other relevant information:

### Additional context
I will try to use GPU instead of TPUs

cc @kaushikb11 @rohitgr7 @awaelchli @ananthsub @ninginthecloud

Start by reproducing BERT-large training on 8 TPU cores in the Kaggle TPU environment and observe memory at the epoch boundary before the second epoch. Record the missing PyTorch Lightning, PyTorch, and installation details, then trace the TPU training path and epoch-end handling. Done means the second epoch starts without the memory blow-up and the regression is covered by a reproducible test.

Written by the indexing model from the issue text.