`log_every_n_steps` is broken for `Trainer.{validate,test,predict}`

Nobody has claimed this yet.

The huge issue today with `log_every_n_steps` is that with high probability, it is broken for Trainer.validate, Trainer.test, and Trainer.predict.

`log_every_n_steps` works with the trainer's `global_step` to determine if data is going to be logged: https://github.com/PyTorchLightning/pytorch-lightning/blob/f9b9cdb0d1d4e26d25fe13f19f12ea88690aa0a8/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py#L74-L77

`global_step` is defined as the number of parameter updates that occur. This number is incremented *only* during fitting. It is *never* incremented for validation, test, or prediction routines.

Therefore, if your LightningModule calls `self.log` inside of validate, test, or predict steps, it is highly likely that your data will **not** be logged! You would have to get lucky that the global step is a multiple of `log_every_n_steps` from a prior fitting routine. It is not obvious at all to an end user why calling `self.log` results in no data being updated. This has also been a consistent complaint amongst users of Lightning that I'm aware of, especially when trying to log metrics with Trainer.test.

The workaround we have is to set `Trainer(log_every_n_steps=1)` and gate `LightningModule.log` with another flag for the log frequency. This way, users can at least control the granularity with which they want to log without any surprising behavior from the Lightning framework interfering with this.

We need to update the logger connector code to take into account the batch_idx if validate, test, or predict were called instead of the global step

_Originally posted by @ananthsub in https://github.com/PyTorchLightning/pytorch-lightning/issues/9726#issuecomment-963764581_

cc @tchaton @carmocca

Start in pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py around lines 74-77, where log_every_n_steps is evaluated against global_step. Trace how validation, test, and prediction provide batch_idx and how self.log reaches the connector. Done means logging in those routines respects the configured frequency without depending on a prior fitting run.

Written by the indexing model from the issue text.