ModelCheckpoint `every_n_train_steps` and Trainer `accumulate_grad_batches` (kinda) don't make sense

Bug description

Let say, we have a dataset with length of 60000 samples. With batch size, the length of dataloader is 938. We set every_n_train_steps=10 for ModelCheckpoint

With accumulate_grad_batches=1 (or no accumulation),
the actual number of training steps is 938. ModelCheckpoint saves file to epoch=0-step=930.ckpt. 930 totally makes sense with 938 because it's (938 // 10) * 10.

Now let set accumulate_grad_batches=8, the actual number of training steps is 938 // 8 or 117. ModelCheckpoint saves file to epoch=0-step=110.ckpt. If someone only looked at the progress bar which shows 938, they would think ModelCheckpoint only save at 110/938 steps.

That being said, 110 is correct because the actual number of training steps is 117. And (117 // 10) * 10 is 110.

It's not really a bug, but it's a confusing behaviour. I think some kind of warning should be printed when accumulate_grad_batches is used.

What version are you seeing the problem on?

v2.0

How to reproduce the bug

import os

import lightning as L
import pandas as pd
import seaborn as sn
import torch
from IPython.display import display
from lightning.pytorch.callbacks import ModelCheckpoint
from torch import nn
from torch.nn import functional as F
from torch.utils.data import DataLoader, random_split
from torchmetrics import Accuracy
from torchvision import transforms
from torchvision.datasets import MNIST

PATH_DATASETS = os.environ.get("PATH_DATASETS", ".")
BATCH_SIZE = 64

class MNISTModel(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)

# Init our model
mnist_model = MNISTModel()

# Init DataLoader from MNIST Dataset
train_ds = MNIST(PATH_DATASETS, train=True, download=True, transform=transforms.ToTensor())
train_loader = DataLoader(train_ds, batch_size=BATCH_SIZE)

# Initialize a trainer
trainer = L.Trainer(
    accelerator="gpu",
    devices=1,
    max_epochs=1,
    accumulate_grad_batches=8,
    callbacks=[
        ModelCheckpoint(every_n_train_steps=10)
    ]
)

# Train the model ⚡
trainer.fit(mnist_model, train_loader)

Error messages and logs

Command input:

tree lightning_logs/

Terminal output:

lightning_logs
└── lightning_logs/version_0
    ├── lightning_logs/version_0/hparams.yaml
    ├── lightning_logs/version_0/events.out.tfevents.1687485462.d26f936af7bf.28.2
    └── lightning_logs/version_0/checkpoints
        └── lightning_logs/version_0/checkpoints/epoch=0-step=110.ckpt

Environment

--2023-06-23 01:36:23-- https://raw.githubusercontent.com/Lightning-AI/lightning/master/requirements/collect_env_details.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2759 (2.7K) [text/plain]
Saving to: ‘collect_env_details.py.1’

collect_env_details 100%[===================>] 2.69K --.-KB/s in 0s

2023-06-23 01:36:23 (36.7 MB/s) - ‘collect_env_details.py.1’ saved [2759/2759]

More info

This Kaggle shall provide everything to reproduce the behaviour.

cc @justusschock @lantiga @borda

Start by running the provided MNIST reproduction with ModelCheckpoint(every_n_train_steps=10) and Trainer(accumulate_grad_batches=8). Inspect how ModelCheckpoint and Trainer count training steps, then determine where the warning should be emitted. Done means the confusing mismatch is clearly communicated without changing the checkpoint schedule, with coverage for accumulated and non-accumulated training.