huggingface / huggingface/lighteval
[BUG] chrF++/chrF/TER metrics receive references in wrong format, causing incorrect corpus-level scoring
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
`CorpusLevelTranslationMetric` passes references to sacrebleu in `[sent_id][ref_id]` shape (per-sample list), but sacrebleu expects `[ref_id][sent_id]` (reference streams). This causes chrF/chrF++/TER to score only the first min number of refs per sample hypotheses and compare them against a pooled set of references from the entire dataset. BLEU is special-cased but still drops extra references.
## To Reproduce
Minimal example showing the shape issue (mirrors how lighteval currently passes refs):
```python
from lighteval.metrics.metrics_corpus import CorpusLevelTranslationMetric
from lighteval.metrics.sample_preparator import GenerativeCorpusMetricInput
from lighteval.utils.utils import as_list
items = [
GenerativeCorpusMetricInput(golds=["GOOD"], preds=["GOOD"]),
GenerativeCorpusMetricInput(golds=["REF2"], preds=["PRED2"]),
]
metric = CorpusLevelTranslationMetric("chrf++")
# Mirrors compute_corpus(): each i.golds is Sequence[str], so this produces
# Sequence[Sequence[str]] in per‑sample orientation.
golds = [i.golds for i in items] # [sent_id][ref_id]
preds = [as_list(i.preds)[0] for i in items]
# Shows only one hypothesis is being scored:
stats = metric.get_metric()._extract_corpus_statistics(preds, golds)
print(len(stats)) # 1 (should be 2)
score_wrong = metric.get_metric().corpus_score(preds, golds).score
print(score_wrong) # 100 despite 2nd hyp being wrong (0 for TER)
```
## Expected behavior
Each hypothesis should be scored against its own reference set, and corpus statistics should include all hypotheses (len(stats) == len(hypotheses)).
## Version info
* lighteval: 0.13.0
* Python: 3.13
* Dependencies: sacrebleu 2.5.1
## Suspected root cause
* `GenerativeCorpusMetricInput.golds` is `list[str]` (per-sample refs) (src/lighteval/metrics/sample_preparator.py).
* `compute_corpus()` does `golds = [i.golds for i in items]`, producing `list[list[str]]`. This type passes, but the orientation is per-sample, not per-reference (src/lighteval/metrics/metrics_corpus.py).
* sacrebleu expects `[ref_id][sent_id]` and builds per-segment refs via `zip(*references)`, then pairs them with hypotheses using `zip(hypotheses, ref_cache)`, truncating to the number of refs (sacrebleu/metrics/base.py).
* chrF++ picks the best ref among those provided (sacrebleu/metrics/chrf.py `_compute_segment_statistics`, where `best_f_score` is updated per ref). The best match is usually the corresponding reference (e.g., ref1 for hyp1) but not necessarily, which can inflate scores. TER uses the same base machinery (sacrebleu/metrics/ter.py + sacrebleu/metrics/base.py).
## Suggested fix
In metrics_corpus.py, `CorpusLevelTranslationMetric` class, transpose `golds` before calling sacrebleu so it matches `[ref_id][sent_id]`:
```python
from itertools import zip_longest # zip_longest to account for variable num of refs per sample
# inside compute_corpus(), before corpus_score(...)
golds = [list(refs) for refs in zip_longest(*golds, fillvalue=None)]
```
We can also consider applying the same transpose for BLEU to keep multi-reference support instead of dropping to `gold[0]`.
If I missed something or this is intended behavior, please let me know.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.