Batch Size Finder: Allow datamodule to decide fate when OOM thrown with new hook

🚀 Feature

Related #10243 (PR did not get to the finish line).

Motivation

After speaking to "Harry" on the Lightning Slack channel, we came up with some improvements that can be made to the batch size finder. This spawned from a discussion around automatic accumulation in real-time, but ended up being closer to the developer experience when using the batch size finder before training.

Pitch

We should allow the DataModule to control what happens when the batch size finder resets the dataloaders. This is required when certain data-loaders need to be reset in a particular manner (order by size of examples, such that the largest examples are first).

class MyDataModule(pl.LightningDataModule)
    ...
    def on_batch_size_finder_reset(self, model, trainer):
        trainer.reset_train_dataloader(model)
        trainer.reset_val_dataloader(model)

This function would be called via the batch size finder when a reset needs to be made, i.e:
https://github.com/Lightning-AI/lightning/blob/master/src/pytorch_lightning/tuner/batch_size_scaling.py#L149-L150

or

https://github.com/Lightning-AI/lightning/blob/master/src/pytorch_lightning/tuner/batch_size_scaling.py#L188-L189

We should also ensure that we reset when an OOM is raised, as this is omitted from the exception clause: https://github.com/Lightning-AI/lightning/blob/master/src/pytorch_lightning/tuner/batch_size_scaling.py#L195

cc @borda @akihironitta @rohitgr7 @carmocca