huggingface / huggingface/evaluate

wrong rouge-Lsum results due to lack of parameter control

Open
#501 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

https://github.com/huggingface/evaluate/blob/937a8ddbd74abaaf0d63b9ff2290fccd0277eca2/metrics/rouge/rouge.py#L132

This line creates a RougeScorer object, which also includes ```split_summaries=False``` in its original implementation.

``` split_summaries``` set to ```False``` splits texts into sentences for ROUGE-LSum based on ```\n```, while setting it to ```True``` allows for a more complex tokenization based on punctuation.

I think it is worth adding control over this parameter to evaluate rouge interface since normally summary sentences aren't separated by newlines and not having control over this parameters leads to ROUGE-L and ROUGE-LSum always producing the same result while being different metrics.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.