Lightning-AI / Lightning-AI/pytorch-lightning

warnings: resuming before epoch end is absolutely normal for long trainings

Open
#18,780 7 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

data handling feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

forking from https://github.com/Lightning-AI/lightning/issues/18723#issuecomment-1751307472 where we were discussing various warnings that don't necessarily apply to all.

This issue discusses this warnings:

```
[...]python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py:151: UserWarning: You're
resuming from a checkpoint that ended before the epoch ended. This can cause unreliable results if
further training is done. Consider using an end-of-epoch checkpoint
```
Many shared SLURM environments have a relatively short time limit to each job so one can't do one epoch w/o restart and resuming. e.g. some have only 20h top.

In "can cause unreliable results" are you perhaps implying that there is no guarantee the DL will not continue from where it left off on saving the last checkpoint but will repeat the same data? Shouldn't PTL save the worker RNG state and correctly restore it on resume? Though with a custom DL there is no way PTL could easily do that.

But in general a 3 months training will take many restarts, not only because of a short SLURM job limit, but also because there will be divergences requiring rollbacks, which means restarts.

And yes the operator needs to be super-aware whether the resume breaks the unique flow of samples and leads to replacements.

cc @borda @justusschock @awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at pytorch_lightning/loops/training_epoch_loop.py:151 and inspect the warning emitted when resuming from a checkpoint that ended before the epoch. Clarify whether the warning should apply to long, repeatedly resumed training and how checkpoint restoration affects worker RNG and sample continuity. Done means an agreed warning behavior supported by verification of the resume semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.