Extra training step/global_step incrementation when resuming training from a checkpoint

Bug description

When resuming training from an end-of-epoch checkpoint, the global_step counter is incremented an additional time before training continues, suggesting that an additional training step is being ran.

In the minimal code example below that can be used to reproduce this

1. the model is trained for 1 epoch over 3 batches
2. then checkpoint
3. resume training from checkpoint, train for another epoch of 3 batches
4. checkpoint
5. resume training from checkpoint, train for another epoch of 3 batches

totaling 9 training steps across the entire training process. However, I am getting the output:

----start----
  mid 	epoch - train - Epoch: 	0 Global step: 	0
  mid 	epoch - train - Epoch: 	0 Global step: 	1
  mid 	epoch - train - Epoch: 	0 Global step: 	2
end 	epoch - valid - Epoch: 	0 Global step: 	3
end 	epoch - train - Epoch: 	0 Global step: 	3
----from checkpoint----
Loading new checkpoint.
  mid 	epoch - train - Epoch: 	0 Global step: 	3
end 	epoch - valid - Epoch: 	0 Global step: 	4
end 	epoch - train - Epoch: 	0 Global step: 	4
  mid 	epoch - train - Epoch: 	1 Global step: 	4
  mid 	epoch - train - Epoch: 	1 Global step: 	5
  mid 	epoch - train - Epoch: 	1 Global step: 	6
end 	epoch - valid - Epoch: 	1 Global step: 	7
end 	epoch - train - Epoch: 	1 Global step: 	7
----from checkpoint----
Loading new checkpoint.
  mid 	epoch - train - Epoch: 	1 Global step: 	7
end 	epoch - valid - Epoch: 	1 Global step: 	8
end 	epoch - train - Epoch: 	1 Global step: 	8
  mid 	epoch - train - Epoch: 	2 Global step: 	8
  mid 	epoch - train - Epoch: 	2 Global step: 	9
  mid 	epoch - train - Epoch: 	2 Global step: 	10
end 	epoch - valid - Epoch: 	2 Global step: 	11
end 	epoch - train - Epoch: 	2 Global step: 	11

suggesting that the model is trained for 11 training steps. However, I am expecting:

----start----
    mid epoch - train - Epoch: 	0 Global step: 	0
    mid epoch - train - Epoch: 	0 Global step: 	1
    mid epoch - train - Epoch: 	0 Global step: 	2
end epoch - valid - Epoch: 	0 Global step: 	3
end epoch - train - Epoch: 	0 Global step: 	3
----from checkpoint----
    mid epoch - train - Epoch: 	1 Global step: 	3
    mid epoch - train - Epoch: 	1 Global step: 	4
    mid epoch - train - Epoch: 	1 Global step: 	5
end epoch - valid - Epoch: 	1 Global step: 	6
end epoch - train - Epoch: 	1 Global step: 	6
----from checkpoint----
    mid epoch - train - Epoch: 	2 Global step: 	6
    mid epoch - train - Epoch: 	2 Global step: 	7
    mid epoch - train - Epoch: 	2 Global step: 	8
end epoch - valid - Epoch: 	2 Global step: 	9
end epoch - train - Epoch: 	2 Global step: 	9

since the model should only be trained for 3 batches per epoch, for 3 epochs, i.e.: 9 batches total. Note the additional calls to hooks training_step, on_train_epoch_end and on_validation_epoch_end with the previous epoch IDs when the training is resumed from a checkpoint, which seems to be causing this issue.

This seems to be similar to #11555 .

What version are you seeing the problem on?

v2.1

How to reproduce the bug

import torch
from pytorch_lightning import LightningModule, LightningDataModule
from pytorch_lightning import Trainer
from torch.utils.data import Dataset, DataLoader
from pytorch_lightning.callbacks import ModelCheckpoint


class CounterModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def training_step(self, batch, batch_idx):
        loss = self.layer(batch).sum()
        print(f'  mid \tepoch - train - Epoch: \t{self.current_epoch} Global step: \t{self.global_step}')
        return loss

    def validation_step(self, batch, batch_idx):
        loss = self.layer(batch).sum()
        return loss

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)
    
    def on_train_epoch_end(self):
        print(f'end \tepoch - train - Epoch: \t{self.current_epoch} Global step: \t{self.global_step}')

    def on_validation_epoch_end(self):
        print(f'end \tepoch - valid - Epoch: \t{self.current_epoch} Global step: \t{self.global_step}')

    def on_load_checkpoint(self, checkpoint):
        print('Loading new checkpoint.')


class RandomDataset(Dataset):
    def __init__(self, size, num_samples):
        self.len = num_samples
        self.data = torch.randn(num_samples, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class RandomDatamodule(LightningDataModule):
    def __init__(self):
        super(RandomDatamodule, self).__init__()

        self.train_dataset = RandomDataset(32, 64)
        self.valid_dataset = RandomDataset(32, 64)

    def train_dataloader(self):
        return DataLoader(dataset=self.train_dataset,
                          batch_size=2,
                          drop_last=False)

    def val_dataloader(self):
        return DataLoader(dataset=self.valid_dataset,
                          batch_size=2,
                          drop_last=False)


checkpoint_callback = ModelCheckpoint(
    dirpath='global_step_test',
    filename=None,
    save_top_k=-1,
    save_last=True,
    every_n_train_steps=1,
    save_on_train_epoch_end=True)


data = RandomDatamodule()
trainer = Trainer(limit_train_batches=3, 
                  max_epochs=1,
                  enable_progress_bar=False, 
                  enable_model_summary=False,
                  callbacks=checkpoint_callback,
                  num_sanity_val_steps=0)
model = CounterModel()
print('----start----')
trainer.fit(model=model,
                    datamodule=data)

print('----from checkpoint----')
trainer = Trainer(limit_train_batches=3, 
                  max_epochs=2, 
                  enable_progress_bar=False, 
                  enable_model_summary=False,
                  callbacks=checkpoint_callback,
                  num_sanity_val_steps=0)
trainer.fit(model=model,
                    datamodule=data,
                    ckpt_path='global_step_test/last.ckpt')
print('----from checkpoint----')
trainer = Trainer(limit_train_batches=3, 
                  max_epochs=3, 
                  enable_progress_bar=False, 
                  enable_model_summary=False,
                  callbacks=checkpoint_callback,
                  num_sanity_val_steps=0)
trainer.fit(model=model,
                    datamodule=data,
                    ckpt_path='global_step_test/last.ckpt')

Error messages and logs

# Error messages and logs here please

Environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info

No response

cc @lantiga

Start with the Trainer.fit checkpoint-resume flow, the ModelCheckpoint configuration, and the LightningModule hooks shown in the reproduction. Compare hook and global_step behavior when resuming from an end-of-epoch checkpoint; done means the example reports 9 training steps without duplicate previous-epoch hook calls.