huggingface / huggingface/evaluate

Integrate `scikit-learn` metrics into `evaluate`

Open
#297 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

## Summary

We want to support the wide range of metrics implemented in `scikit-learn` in `evaluate`. While this expands the capabilities of `evaluate` it also gives users from the `scikit-learn` ecosystem access to useful tools in `evaluate` such as pushing results to the hub or end-to-end evaluate models with the `evaluator` classes. As a bonus, all metrics will get an interactive widget that can be embedded in various places such as the docs.

## Goal

The goal of this integration should be that metrics from `scikit-learn` can be loaded from evaluate with the following API:

```python
import evaluate

metric = evaluate.load("sklearn/accuracy")
metric.compute(predictions=[0, 1, 1], references=[1, 1, 0])
```

## How it can be done

For the integration we could build a script that goes through all metrics of the `scikit-learn` repository and automatically builds the metric repositories in the `evaluate` format and pushes them to the Hub. This could be a script that's executed via a GitHub action whenever a change is pushed to main similar to how it's done for the internal modules (see [here](https://github.com/huggingface/evaluate/blob/main/.github/hub/push_evaluations_to_hub.py)).

Besides the function, its arguments and input/output format we can also use the docs to populate the `gradio` widget on the hub. See the [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) module as an example of how the metrics could be displayed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.