huggingface / huggingface/evaluate
METEOR has no option to return unaggregated results
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
## Describe the bug
The METEOR metric returns the mean metric as opposed to a list of individual metrics per ref/pred pair. This is inconsistent with other default metric behavior including rouge, bleu, and bertscore. This can be a problem when trying to calculate correlation, for example.
See `metrics/meteor/meteor.py` line 168 in `_compute`:
`return {"meteor": np.mean(scores)}`
## Steps to reproduce the bug
```python
from evaluate import load
metric = load("meteor")
meteor.compute(references=["reference one", "reference two"], predictions=["prediction one", "prediction two"])
```
## Expected results
`{"meteor": [0.25, 0.25]}`
## Actual results
`{"meteor": 0.25}`
## Environment info
evaluate version: 0.4.1
Platform: Rocky Linux 8.9
Python version: 3.9.18
Contributor guide
Assessment
This issue has not been assessed yet.