huggingface / huggingface/evaluate

Feature: integration standard libraries

Open
#11 6 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Create a uniform integration for metrics libraries such as `scikit-learn`, `keras.metrics` or `torchmetrics` instead of integrating them metrics from these frameworks one-by-one. The goal of the integration is to be able to load the metrics from these frameworks with a syntax as follows:

```
accuracy = load_metric("scikit-learn/accuracy")
```

This way there is also more transparency where the metrics come from especially if they are just a wrapper around existing libraries. This would also simplify maintaining them by avoiding the need to change every metric separately if needed.

Popular metrics libraries:
- [ ] [`scikit-learn`](https://scikit-learn.org/stable/modules/model_evaluation.html)
- [ ] [`keras.metrics`](https://keras.io/api/metrics/)
- [ ] [`torchmetrics`](https://torchmetrics.readthedocs.io/en/latest/)

Others (maybe a bit more niche):
- [ ] [`PyDML`](https://pydml.readthedocs.io/en/latest/)
- [ ] [`networkX`](https://networkx.org/documentation/stable/reference/algorithms/similarity.html)
- [ ] [`MLFlow`](https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.evaluate)

If all metrics (incl. "canonical" ones) will be on the hub such an integration could just be a script that automatically creates (or updates) the necessary repositories on the hub.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.