huggingface / huggingface/evaluate

Evaluator needs a simple way to check if a specific model is supported

Open
#343 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

With @sashavor, we found that the below example doesn't work with the QA Evaluator — but it's not obvious that it wouldn't.

Docs for the QA Evaluator say "Methods in this class assume a data format compatible with the [QuestionAnsweringPipeline](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.QuestionAnsweringPipeline)." That links out to https://huggingface.co/models?pipeline_tag=question-answering and searching for "t5" returns "t5-base-finetuned-quartz", but that model is incompatible with the QA pipeline and it's not immediately clear that it wouldn't work unless you actually tried it.

It's probably not feasible to list all models on the Hub which are compatible with any given evaluator, but maybe it'd be useful to have a helper method which checks if a model is compatible with an Evaluator ahead of time before setting an entire pipeline? Welcoming suggestions on how we could handle this!

```
from datasets import load_dataset
from evaluate import evaluator

task_evaluator = evaluator("question-answering")

data = load_dataset("squad", split="validation[:10]")
eval_results = task_evaluator.compute(
model_or_pipeline="mrm8488/t5-base-finetuned-quartz",
data=data,
metric="squad",
strategy="bootstrap",
n_resamples=30
)

print(eval_results)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.