open-mmlab / open-mmlab/mmengine

Add documentation of evaluation on multiple dataset with multiple metric.

Open
#176 0 comments 1 reaction 1 assignee View on GitHub

@Harold-lkk is already working on this.

Since May 23, 2022.

planned feature
Dominant language
Python
Stars
1.5k
Forks
464
PR merge metrics
No merged PRs in 30d

Description

Describe the feature
Add documentation to show how to evaluate multiple datasets with multiple metrics and use one of the metrics of a dataset as the best indicator.

Motivation
Users might need to evaluate different metrics on multiple datasets.
In such a case, only one metric on one dataset needs to be selected to indicate whether the model is the best model and should be saved.
It is unnecessary to officially support this feature in MMEngine, but MMEngine supports users to create a new Loop class to support this feature. Therefore, we should update the documentation to show such an example.

Related resources
See a previous PR in MMSeg https://github.com/open-mmlab/mmsegmentation/pull/1461

Additional context
Add any other context or screenshots about the feature request here.
If you would like to implement the feature and create a PR, please leave a comment here and that would be much appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.