huggingface / huggingface/evaluate

BLEU: improve input validation and document edge-case behavior)

Open
#729 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Hi! I noticed a few small, backward-compatible improvements that could clarify and harden the BLEU metric implementation.

- Support a simple string alias for the default tokenizer (e.g. `tokenizer="13a"`) in addition to passing a callable.
- Add explicit validation for length mismatches between `predictions` and `references`.
- Document and add tests for the current behavior when predictions are empty strings (BLEU evaluates to 0.0 implicitly today).

These changes don’t alter default behavior and aim to improve usability, robustness, and reproducibility.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.