google-research / google-research/bleurt
Results mismatch using released BLEURT-Large-128
- Dominant language
- Python
- Stars
- 793
- Forks
- 94
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
Recently I'm interested in reimplementing and investigating your research. However, when I directly use your released code and BLEURT-Large-128 checkpoint model, I can't get comparable results with what you present [here](https://github.com/google-research/bleurt/blob/master/checkpoints.md). Here is what I got:
- de-en: 29.15
- fi-en: 30.64
- gu-en: 27.49
- kk-en: 39.14
- lt-en: 34.28
- ru-en: 26.75
- zh-en: 41.98
- avg: 32.78
I first follow your command to get the WMT2019 data and the BLEURT-Large-128 checkpoint. After evaluating the whole dataset file, I collect the prediction scores, split the results and corresponding golden scores by language pairs, and compute Kendall results using scipy.stats.kendalltau as your implementation.
So I'm wondering I've missed any detail. Could you help me? Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.