huggingface / huggingface/evaluate

Compute BLEU score of a Pandas DataFrame with valid rows filtered

Open
#469 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

I have a Pandas DataFrame from an Excel file, which contains text data which need to calculate the BLEU score row-by-row.

```
import evaluate
import pandas as pd
sacrebleu = evaluate.load("sacrebleu")

testset = pd.read_excel(xlsx_filename)
# find out valid rows with all columns are valid
valid_rows = testset['col1'].notna() & testset['col2'].notna() & testset['col3'].notna()

for i in range(len(testset)): # or... for i in range(len(testset.loc[valid_rows, 'col2']))
score = sacrebleu.compute(predictions=[testset.loc[valid_rows, 'col1'][i], testset.loc[valid_rows, 'col2'][i]], references=[testset.loc[valid_rows, 'col3'][i]])
```

It raises `KeyError: 139`.

The length of `valid_rows` and `testset` are 13700, while the length of `testset.loc[valid_rows, 'col2']` is 12208.

I know loop through for-loop is an anti-pattern, but how can I fit a Series into the `sacrebleu.compute()` function? It accepts only `[string, string], string` as input.

How can I solve this problem?

---
This question is also at: https://stackoverflow.com/questions/76581453/compute-bleu-score-of-a-pandas-dataframe-with-valid-rows-filtered

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.