huggingface / huggingface/evaluate
Allow BLEURT optimisation parameters
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
BLEURT is very slow to compute: I'm observing 426.62 seconds on a g5.xlarge with `BLEURT-20-D6` for 1459 sentences.
It'd be useful to be able to use the optimisations specified in [speeding-up-bleurt](https://github.com/google-research/bleurt/tree/master#speeding-up-bleurt)
1. Tune batch size by passing `batch_size` as a kwarg to `scorer.score` (https://github.com/google-research/bleurt/blob/cebe7e6f996b40910cfaa520a63db47807e3bf5c/bleurt/score.py#L175)
2. Use the `LengthBatchingBleurtScorer` (https://github.com/google-research/bleurt/blob/cebe7e6f996b40910cfaa520a63db47807e3bf5c/bleurt/score.py#L227C7-L227C33)
Contributor guide
Assessment
This issue has not been assessed yet.