huggingface / huggingface/evaluate
Bug in BLEU calculation - BLEU of identical strings should be 1.0, not 0.0
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
I think I discovered a bug related to BLEU calculation which occurs if you compare two identical strings. The expected behavior is that the BLEU of completely identical strings is 1.0, while in some cases I get 0.0.
## OS and Python versions
- **OS:** Windows 10 Pro
- **Python:** 3.10.13
- **Evaluate:** 0.4.1
## Bug description
If you run:
```
import evaluate
bleu = evaluate.load("bleu")
bleu.compute(predictions=["Duplicate string"], references=[["Dupliacte string"]])
```
you get:
```
{'bleu': 0.0,
'precisions': [0.5, 0.0, 0.0, 0.0],
'brevity_penalty': 1.0,
'length_ratio': 1.0,
'translation_length': 2,
'reference_length': 2}
```
as the output. I expected the `'bleu'` to be `1.0`.
Similar thing happens if you run:
`bleu.compute(predictions=["foobar"], references=[["foobar"]])`
you get:
```
{'bleu': 0.0,
'precisions': [1.0, 0.0, 0.0, 0.0],
'brevity_penalty': 1.0,
'length_ratio': 1.0,
'translation_length': 1,
'reference_length': 1}
```
Again, I expected BLEU to be `1.0`, not `0.0`.
Contributor guide
Assessment
This issue has not been assessed yet.