huggingface / huggingface/evaluate
Feature: integration standard libraries
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Create a uniform integration for metrics libraries such as `scikit-learn`, `keras.metrics` or `torchmetrics` instead of integrating them metrics from these frameworks one-by-one. The goal of the integration is to be able to load the metrics from these frameworks with a syntax as follows:
```
accuracy = load_metric("scikit-learn/accuracy")
```
This way there is also more transparency where the metrics come from especially if they are just a wrapper around existing libraries. This would also simplify maintaining them by avoiding the need to change every metric separately if needed.
Popular metrics libraries:
- [ ] [`scikit-learn`](https://scikit-learn.org/stable/modules/model_evaluation.html)
- [ ] [`keras.metrics`](https://keras.io/api/metrics/)
- [ ] [`torchmetrics`](https://torchmetrics.readthedocs.io/en/latest/)
Others (maybe a bit more niche):
- [ ] [`PyDML`](https://pydml.readthedocs.io/en/latest/)
- [ ] [`networkX`](https://networkx.org/documentation/stable/reference/algorithms/similarity.html)
- [ ] [`MLFlow`](https://www.mlflow.org/docs/latest/python_api/mlflow.html#mlflow.evaluate)
If all metrics (incl. "canonical" ones) will be on the hub such an integration could just be a script that automatically creates (or updates) the necessary repositories on the hub.
Contributor guide
Assessment
This issue has not been assessed yet.