Lightning-AI / Lightning-AI/pytorch-lightning

Add `min_delta` argument to `ModelCheckpoint` callback

Open
#14,353 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

callback: model checkpoint feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

Unlike the `ModelCheckpoint` callback `EarlyStopping` has the `min_delta` argument, which controls the criteria for training stopping, I.e how much the monitored value must be improved to continue further training. Such behavior provides you with minimal regularization to your model and saves some time because you wouldn't continue training if your model improves a little bit, e.g +0.0001, which sign of overfitting to the validation dataset.

However, when using `ModelCheckpoint` with or without `EarlyStopping` I can't control this behavior and maybe my best checkpoint will be rewritten with the overfitted model.

This problem will be really strange in research.

### Motivation

The motivation of this proposal is to reduce the chance of overfitting the validation dataset. I didn't meet similar issues like this one before.

### Pitch

I suppose to add a new argument to the `ModelCheckpoint`/for controlling the saving checkpoint relatively on how much the monitored value was improved.

### Alternatives

Unfortunately, I don't see any alternatives.

### Outline

Also, I would to join one team to write the code of `ModelCheckpoint` for the `min_delta` argument, of course, if the issue will be agreed upon.

cc @borda @carmocca @awaelchli @ninginthecloud @jjenniferdai @rohitgr7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the ModelCheckpoint and EarlyStopping callback implementations and comparing how min_delta is defined and applied. Confirm the intended checkpoint-improvement behavior, then add the argument and tests showing that tiny monitored-value changes do not replace the best checkpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.