huggingface / huggingface/evaluate

Evaluation not working with `microsoft/deberta-v3-large`

Open
#553 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Hi,

I was trying to evaluate the model trained on `microsoft/deberta-v3-large` but looks like it's having some issues on indexing. Any ideas or support for this one ?

Error:
`IndexError: list index out of range`

Code to Reproduce:
```
import pandas as pd
from datasets import load_dataset
from evaluate import evaluator
from transformers import pipeline

models = [
"microsoft/deberta-v3-large"
]

data = load_dataset("conll2003", split="validation").shuffle().select(range(10))
task_evaluator = evaluator("token-classification")

results = []
for model in models:
results.append(
task_evaluator.compute(
model_or_pipeline=model, data=data, metric="seqeval"
)
)

df = pd.DataFrame(results, index=models)
df[["overall_f1", "overall_accuracy", "total_time_in_seconds", "samples_per_second", "latency_in_seconds"]]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.