Lightning-AI / Lightning-AI/pytorch-lightning

Add `best_model_metrics` to `ModelCheckpoint` callback

Open
#19,007 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

callback: model checkpoint feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

There doesn't seem to be a straightforward way to get the training and validation metrics of the best model as monitored by ModelCheckpoint. Adding `best_model_metrics` attribute to `ModelCheckpoint` can be done easily in a way similar to how `best_model_score` works now.

### Pitch

Accessing training and validation metrics of the best model checkpoint by accessing `trainer.checkpoint_callback.best_model_metrics` right after a `trainer.fit` call is useful for hyperparameter optimization if you want to optimize based on a validation metric or use other custom callbacks to summarize a batch experiment. I am willing to implement this feature.

### Alternatives

1. Running `trainer.test` on the training and validation dataset again after a `trainer.fit` call.
2. Getting records saved by loggers by checking the epoch number in the filename of a checkpoint file.

These two options create unnecessary overheads for a simple feature like this.

### Additional context

https://github.com/Lightning-AI/lightning/discussions/18398

cc @lantiga @borda

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ModelCheckpoint callback and inspect how best_model_score is maintained. Use trainer.fit as the entry point and verify that trainer.checkpoint_callback.best_model_metrics exposes the training and validation metrics associated with the best checkpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.