Decide what to do about 16bit weights trained with mixed precision

Nobody has claimed this yet.

Our training scripts select mixed precision by default (16-mixed or bf16-mixed).
Many of the HF pretrained weights come in 16bit (float16 or bfloat16).

Since the weights are already in this dtype, it's not useful to do mixed precision training. We can do one of two things about it:

a) Cast the weights to fp32 in this case so that mixed precision training does something useful
b) Raise an exception saying that this configuration is not useful.

Note that this change needs to be done in PyTorch Lightning and Fabric, but I'm opening the issue here because this is a common problem in LitGPT and because we would need to add code to LitGPT to avoid b) if we choose it

No specific files or tests are named. Start with the training scripts' mixed-precision setup and review how PyTorch Lightning and Fabric handle 16-bit weights; done requires choosing a policy, implementing it across the affected projects, and ensuring LitGPT remains compatible.

Written by the indexing model from the issue text.