Lightning-AI / Lightning-AI/pytorch-lightning
Add `min_delta` argument to `ModelCheckpoint` callback
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
Unlike the `ModelCheckpoint` callback `EarlyStopping` has the `min_delta` argument, which controls the criteria for training stopping, I.e how much the monitored value must be improved to continue further training. Such behavior provides you with minimal regularization to your model and saves some time because you wouldn't continue training if your model improves a little bit, e.g +0.0001, which sign of overfitting to the validation dataset.
However, when using `ModelCheckpoint` with or without `EarlyStopping` I can't control this behavior and maybe my best checkpoint will be rewritten with the overfitted model.
This problem will be really strange in research.
### Motivation
The motivation of this proposal is to reduce the chance of overfitting the validation dataset. I didn't meet similar issues like this one before.
### Pitch
I suppose to add a new argument to the `ModelCheckpoint`/for controlling the saving checkpoint relatively on how much the monitored value was improved.
### Alternatives
Unfortunately, I don't see any alternatives.
### Outline
Also, I would to join one team to write the code of `ModelCheckpoint` for the `min_delta` argument, of course, if the issue will be agreed upon.
cc @borda @carmocca @awaelchli @ninginthecloud @jjenniferdai @rohitgr7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the ModelCheckpoint and EarlyStopping callback implementations and comparing how min_delta is defined and applied. Confirm the intended checkpoint-improvement behavior, then add the argument and tests showing that tiny monitored-value changes do not replace the best checkpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100