huggingface / huggingface/evaluate

Feature: standardize inputs/outputs of metrics

Open
#10 28 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Currently there are several different inputs/output formats possible in `Metrics`. We should standardize them as much as possible and respecting the following principle:

- inputs/outputs are easy to understand and use
- outputs are compatible with other frameworks

For the output standardization: probably a dictionary structure, even if nested would be ok. Also a dedicated output class could be considered like in `transformer` models but this is probably not necessary here. To make it compatible with e.g. `keras` we could add a postprocess function at initialization similar a `transform` in `datasets`.

There are three options we could implement:
```Python
load_metric(..., postprocess="metric_key") # equivalent result to `metric.compute()["metric_key"]`
load_metric(..., postprocess="flatten") # equivalent flattening the output dict: `flatten(metric.compute())`
load_metric(..., postprocess=func) # equivalent result to `func(metric.compute())`
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.