huggingface / huggingface/disaggregators

Set up simple intersectional disaggregated evaluation through Evaluate

Open
#7 1 comment 0 reactions 1 assignee Claimed by @NimaBoscarino View on GitHub
enhancement
Dominant language
Python
Stars
70
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Required:

* A sample script showing how to take a HF dataset, disaggregate it, run an Evaluator on it, and display the resulting evaluation metrics disaggregated by a combination of disaggregation modules.
* Ideally this should be built without having to fork Evaluate. If a custom Evaluator is required, it can be added to this repo and the `evaluate` dependency can be added as an extra (optional) dependency to this library.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.