huggingface / huggingface/evaluate
Feature: standardize inputs/outputs of metrics
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Currently there are several different inputs/output formats possible in `Metrics`. We should standardize them as much as possible and respecting the following principle:
- inputs/outputs are easy to understand and use
- outputs are compatible with other frameworks
For the output standardization: probably a dictionary structure, even if nested would be ok. Also a dedicated output class could be considered like in `transformer` models but this is probably not necessary here. To make it compatible with e.g. `keras` we could add a postprocess function at initialization similar a `transform` in `datasets`.
There are three options we could implement:
```Python
load_metric(..., postprocess="metric_key") # equivalent result to `metric.compute()["metric_key"]`
load_metric(..., postprocess="flatten") # equivalent flattening the output dict: `flatten(metric.compute())`
load_metric(..., postprocess=func) # equivalent result to `func(metric.compute())`
```
Contributor guide
Assessment
This issue has not been assessed yet.