huggingface / huggingface/evaluate

Support for disaggregated evaluations in Evaluators

Open
#406 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

For cases where we want to compute metrics across several groups using the Evaluators, the option is to call `.compute` with different splits of data. This is okay if the groups are disjoint, but for groups that have any overlap that means that we have to recompute inferences (which can be really costly) since there is no caching mechanism.

Maybe a working solution could be to add something like a `disaggregate_by` flag to the `.compute` for the Evaluators (or maybe even to the base evaluator) which could be used to calculate the metric on the specified folds of data _after_ the inferences have been computed.

I made a custom evaluator for something like this while working on [Disaggregators](https://github.com/huggingface/disaggregators) here: [Google Colab](https://colab.research.google.com/drive/1R92qDm0V0nlJJHreUjXbdMhj49DWOrBi#scrollTo=3fsKJeVUa6A0&line=108&uniqifier=1)

With the implementation above, the dataset is expected to have boolean columns like `pronouns.he_him `, `pronouns.they_them`, `pronouns.she_her`. Then, that list of column names is passed to `disaggregate_by` to compute disaggregated evaluations.

Additionally, it's often valuable to combine disaggregations and run metrics on intersectional groups, and I've also got that implemented in that custom evaluator in the notebook above.

Would there be any interest in something like this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.