open-mmlab / open-mmlab/mmengine
Add documentation of evaluation on multiple dataset with multiple metric.
@Harold-lkk is already working on this.
Since May 23, 2022.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 464
- PR merge metrics
- No merged PRs in 30d
Description
Describe the feature
Add documentation to show how to evaluate multiple datasets with multiple metrics and use one of the metrics of a dataset as the best indicator.
Motivation
Users might need to evaluate different metrics on multiple datasets.
In such a case, only one metric on one dataset needs to be selected to indicate whether the model is the best model and should be saved.
It is unnecessary to officially support this feature in MMEngine, but MMEngine supports users to create a new Loop class to support this feature. Therefore, we should update the documentation to show such an example.
Related resources
See a previous PR in MMSeg https://github.com/open-mmlab/mmsegmentation/pull/1461
Additional context
Add any other context or screenshots about the feature request here.
If you would like to implement the feature and create a PR, please leave a comment here and that would be much appreciated.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.