huggingface / huggingface/evaluate

combined metrics in evaluators/Suite subtasks

Open
#444 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

I see that custom evaluators are already discussed in #367, however I don't see a proper way to have multiple metrics for a SubTask and it's evaluator. Apart from writing a custom metric that includes both metrics. Adding the Subtask twice with distinct metrics would take twice as long as all predictions have to be done again.

Since CombinedEvaluations is not an EvaluationModule you can't pass it to an Evaluator. So a lazy solution I came up with is to always do combine() since it should work with a single metric too.

Is there any smarter ways to do this or should I try myself with a PR?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.