Make MiscofigurationException optional for ModelCheckpoint

Description & Motivation

At the moment, the checkpointing subsystem can be very brittle. Suppose the model logs various metrics at different stages of the fit loop, and there are several ModelCheckpoints that monitor these metrics. When the checkpoints get recorded at the end of the epoch, unless all the metrics have been populated, a MisconfigurationException is thrown. This behaviour has already caused complaints in #17150 .

I have stumbled upon this when restoring my model from a checkpoint, although this could be a bug in Lightning version 2.0.1. Somehow, when restoring, the trainer runs part of validation loop for the last epoch, and my model code is written in such a way that only part of the metrics get computed, and the loop crashes.

It would be great if ModelCheckpoint had a constructor parameter boolean strict, that would turn off the exception and write a warning instead.

Pitch

Introduce a constructor parameter in ModelCheckpoint that would stop MiscofigurationException from being thrown when the metric has not been computed, and raise a warning instead. Something like

class ModelCheckpoint:
    def __init__(
        self,
        ...,
        strict: bool = True
   )
"""
Args:
    strict: throw a ``MiscofigurationException`` if the monitored metric has not been computed. 
        Default: ``True``. Set to ``False`` to raise a warning instead.
"""

Alternatives

One alternative is make warnings the default behaviour, so strict=False by default.

Additional context

No response

cc @borda @carmocca @awaelchli

Start at the ModelCheckpoint constructor and the checkpointing path that raises MisconfigurationException when a monitored metric is missing. Add the requested strict option while preserving the current default, and verify that missing metrics warn when strict is false but still raise when it is true, including during checkpoint restoration.