huggingface / huggingface/evaluate

Bug in BLEU calculation - BLEU of identical strings should be 1.0, not 0.0

Open
#529 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

## Summary

I think I discovered a bug related to BLEU calculation which occurs if you compare two identical strings. The expected behavior is that the BLEU of completely identical strings is 1.0, while in some cases I get 0.0.

## OS and Python versions

- **OS:** Windows 10 Pro
- **Python:** 3.10.13
- **Evaluate:** 0.4.1

## Bug description

If you run:

```
import evaluate
bleu = evaluate.load("bleu")
bleu.compute(predictions=["Duplicate string"], references=[["Dupliacte string"]])
```

you get:

```
{'bleu': 0.0,
'precisions': [0.5, 0.0, 0.0, 0.0],
'brevity_penalty': 1.0,
'length_ratio': 1.0,
'translation_length': 2,
'reference_length': 2}
```

as the output. I expected the `'bleu'` to be `1.0`.

Similar thing happens if you run:

`bleu.compute(predictions=["foobar"], references=[["foobar"]])`

you get:

```
{'bleu': 0.0,
'precisions': [1.0, 0.0, 0.0, 0.0],
'brevity_penalty': 1.0,
'length_ratio': 1.0,
'translation_length': 1,
'reference_length': 1}
```

Again, I expected BLEU to be `1.0`, not `0.0`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.