Lightning-AI / Lightning-AI/pytorch-lightning
Allow returning of test results from Trainer.test
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
Currently, `Trainer.test` only returns logged metrics, this is very limiting.
### Motivation
I have a lot of different results which are produced by tests, including figures and sample predictions.
At present, I cannot return them from test since it only allows returning of logged values which it seems must be scalars.
As a workaround I have to separately predict outputs, get targets and then match them together, which is bug-prone.
### Pitch
`Trainer.test` should return output of `Module.test_epoch_end`.
### Alternatives
N/A
### Additional context
N/A
cc @borda @tchaton @justusschock @awaelchli @rohitgr7
Contributor guide
Research direction
Start by tracing the Trainer.test entry point and the Module.test_epoch_end hook to understand how test outputs are currently handled. Review existing Trainer.test tests if available, then verify that returned results include the output from test_epoch_end rather than only logged scalar metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100