huggingface / huggingface/evaluate

[Question]Shall we adding a faster BLEU score calculator?

Open
#586 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

## Shall we?
Hi, I found the [performance request](https://huggingface.co/spaces/evaluate-metric/bleu/discussions/8) about the bleu score. And I'm wondering if we should add a `fast-bleu` or replace the `bleu` implementation with a faster one, like [bleuscore](https://github.com/shenxiangzhuang/bleuscore) which is aimed for better performance for large data size.

## About bleuscore library
### Correctness
I implement bleuscore library with exactly same way as evaluate library and do [serious test with hypothesis](https://github.com/shenxiangzhuang/bleuscore/blob/master/tests/test_bleu_score.py) to make sure they return same result.

### Performance
I wrote the library in Rust and used multi-threading for parallel execution, so it has better performance when the data size gets larger, you can check the [benchmark](https://github.com/shenxiangzhuang/bleuscore?tab=readme-ov-file#benchmark) section for more details.


Benchmark

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.