Lightning-AI / Lightning-AI/pytorch-lightning

Allow returning of test results from Trainer.test

Open
#7,653 11 comments 0 reactions 0 assignees View on GitHub
design feature trainer: test trainer: validate
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature
Currently, `Trainer.test` only returns logged metrics, this is very limiting.

### Motivation

I have a lot of different results which are produced by tests, including figures and sample predictions.
At present, I cannot return them from test since it only allows returning of logged values which it seems must be scalars.

As a workaround I have to separately predict outputs, get targets and then match them together, which is bug-prone.

### Pitch

`Trainer.test` should return output of `Module.test_epoch_end`.

### Alternatives

N/A

### Additional context

N/A

cc @borda @tchaton @justusschock @awaelchli @rohitgr7

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Trainer.test entry point and the Module.test_epoch_end hook to understand how test outputs are currently handled. Review existing Trainer.test tests if available, then verify that returned results include the output from test_epoch_end rather than only logged scalar metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.