huggingface / huggingface/evaluate
wrong rouge-Lsum results due to lack of parameter control
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/huggingface/evaluate/blob/937a8ddbd74abaaf0d63b9ff2290fccd0277eca2/metrics/rouge/rouge.py#L132
This line creates a RougeScorer object, which also includes ```split_summaries=False``` in its original implementation.
``` split_summaries``` set to ```False``` splits texts into sentences for ROUGE-LSum based on ```\n```, while setting it to ```True``` allows for a more complex tokenization based on punctuation.
I think it is worth adding control over this parameter to evaluate rouge interface since normally summary sentences aren't separated by newlines and not having control over this parameters leads to ROUGE-L and ROUGE-LSum always producing the same result while being different metrics.
Contributor guide
Assessment
This issue has not been assessed yet.