Lightning-AI / Lightning-AI/pytorch-lightning
min_epochs and EarlyStopping in conflict
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
I have a problem where I use `min_epochs` because it can take a while before the training starts to converge.
EarlyStopping is triggered quite early, but I thought to set `min_epochs` appropriately to 'get over' that initial period.
However, even though training is converging by the time we reach `min_epochs`, early stopping will stop training immediately once we reached `min_epochs`, just because it was triggered very early on in training.
I think that `EarlyStopping` should pick itself back up if we improve upon the monitored metric before reaching `min_epochs`.
Example `Trainer` config:
```
trainer = L.Trainer(
max_epochs=10000,
callbacks=[
EarlyStopping(monitor="val_loss", mode="min", patience=100),
]
min_epochs=1000,
)
```
Now imagine `EarlyStopping` triggering at epoch 100, but `val_loss` improving at 101 all the way until epoch 1000 - right now training will still stop.
### What version are you seeing the problem on?
v2.2
### How to reproduce the bug
_No response_
### Error messages and logs
_No response_
### Environment
_No response_
### More info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Trainer's min_epochs handling and the EarlyStopping callback, using the configuration in the report as a minimal reproduction. Verify that improvement in val_loss before epoch 1000 prevents stopping at min_epochs, and add regression coverage for that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100