huggingface / huggingface/evaluate
Integrate `scikit-learn` metrics into `evaluate`
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
We want to support the wide range of metrics implemented in `scikit-learn` in `evaluate`. While this expands the capabilities of `evaluate` it also gives users from the `scikit-learn` ecosystem access to useful tools in `evaluate` such as pushing results to the hub or end-to-end evaluate models with the `evaluator` classes. As a bonus, all metrics will get an interactive widget that can be embedded in various places such as the docs.
## Goal
The goal of this integration should be that metrics from `scikit-learn` can be loaded from evaluate with the following API:
```python
import evaluate
metric = evaluate.load("sklearn/accuracy")
metric.compute(predictions=[0, 1, 1], references=[1, 1, 0])
```
## How it can be done
For the integration we could build a script that goes through all metrics of the `scikit-learn` repository and automatically builds the metric repositories in the `evaluate` format and pushes them to the Hub. This could be a script that's executed via a GitHub action whenever a change is pushed to main similar to how it's done for the internal modules (see [here](https://github.com/huggingface/evaluate/blob/main/.github/hub/push_evaluations_to_hub.py)).
Besides the function, its arguments and input/output format we can also use the docs to populate the `gradio` widget on the hub. See the [Accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) module as an example of how the metrics could be displayed.
Contributor guide
Assessment
This issue has not been assessed yet.