huggingface / huggingface/evaluate
Evaluator needs a simple way to check if a specific model is supported
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
With @sashavor, we found that the below example doesn't work with the QA Evaluator — but it's not obvious that it wouldn't.
Docs for the QA Evaluator say "Methods in this class assume a data format compatible with the [QuestionAnsweringPipeline](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.QuestionAnsweringPipeline)." That links out to https://huggingface.co/models?pipeline_tag=question-answering and searching for "t5" returns "t5-base-finetuned-quartz", but that model is incompatible with the QA pipeline and it's not immediately clear that it wouldn't work unless you actually tried it.
It's probably not feasible to list all models on the Hub which are compatible with any given evaluator, but maybe it'd be useful to have a helper method which checks if a model is compatible with an Evaluator ahead of time before setting an entire pipeline? Welcoming suggestions on how we could handle this!
```
from datasets import load_dataset
from evaluate import evaluator
task_evaluator = evaluator("question-answering")
data = load_dataset("squad", split="validation[:10]")
eval_results = task_evaluator.compute(
model_or_pipeline="mrm8488/t5-base-finetuned-quartz",
data=data,
metric="squad",
strategy="bootstrap",
n_resamples=30
)
print(eval_results)
```
Contributor guide
Assessment
This issue has not been assessed yet.