huggingface / huggingface/evaluate

METEOR has no option to return unaggregated results

Open
#572 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug

The METEOR metric returns the mean metric as opposed to a list of individual metrics per ref/pred pair. This is inconsistent with other default metric behavior including rouge, bleu, and bertscore. This can be a problem when trying to calculate correlation, for example.

See `metrics/meteor/meteor.py` line 168 in `_compute`:
`return {"meteor": np.mean(scores)}`

## Steps to reproduce the bug

```python
from evaluate import load

metric = load("meteor")
meteor.compute(references=["reference one", "reference two"], predictions=["prediction one", "prediction two"])
```
## Expected results

`{"meteor": [0.25, 0.25]}`
## Actual results

`{"meteor": 0.25}`

## Environment info

evaluate version: 0.4.1
Platform: Rocky Linux 8.9
Python version: 3.9.18

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.