Save training metadata with the fault tolerance checkpoint

Nobody has claimed this yet.

## 🚀 Feature

See title

### Motivation

Since we will only guarantee fault-tolerance restart for the same number of GPUs and workers (among others), we might want to save metadata about those for archival and error checking.

### Pitch

Add extra fields to the checkpoint generated in `_on_exception`

https://github.com/PyTorchLightning/pytorch-lightning/blob/9d62f248476c6358d8707188f7b20fafa79f8a4f/pytorch_lightning/trainer/trainer.py#L1376-L1381

### Additional context

> This kind of training "metadata" should get saved with the checkpoint. For example, we will also want to know this for fault-tolerance to fail if the trainer configuration has changed between runs and the user is trying to restore mid-batch.

_Originally posted by @carmocca in https://github.com/PyTorchLightning/pytorch-lightning/pull/8515#r677317201_

______________________________________________________________________

#### If you enjoy Lightning, check out our other projects! ⚡

- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.

- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

Start in pytorch_lightning/trainer/trainer.py at the `_on_exception` checkpoint generation referenced by the issue. Determine which training metadata about GPUs, workers, and trainer configuration must be stored for fault-tolerance validation. Done means the checkpoint includes the agreed metadata and supports checking it when restoring mid-batch; the issue does not name tests to run.

Written by the indexing model from the issue text.