Lightning-AI / Lightning-AI/pytorch-lightning

min_epochs and EarlyStopping in conflict

Open
#19,966 3 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug callback: early stopping
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I have a problem where I use `min_epochs` because it can take a while before the training starts to converge.
EarlyStopping is triggered quite early, but I thought to set `min_epochs` appropriately to 'get over' that initial period.
However, even though training is converging by the time we reach `min_epochs`, early stopping will stop training immediately once we reached `min_epochs`, just because it was triggered very early on in training.

I think that `EarlyStopping` should pick itself back up if we improve upon the monitored metric before reaching `min_epochs`.

Example `Trainer` config:

```
trainer = L.Trainer(
max_epochs=10000,
callbacks=[
EarlyStopping(monitor="val_loss", mode="min", patience=100),
]
min_epochs=1000,
)
```
Now imagine `EarlyStopping` triggering at epoch 100, but `val_loss` improving at 101 all the way until epoch 1000 - right now training will still stop.

### What version are you seeing the problem on?

v2.2

### How to reproduce the bug

_No response_

### Error messages and logs

_No response_

### Environment

_No response_

### More info

_No response_

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Trainer's min_epochs handling and the EarlyStopping callback, using the configuration in the report as a minimal reproduction. Verify that improvement in val_loss before epoch 1000 prevents stopping at min_epochs, and add regression coverage for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.