google-research / google-research/bleurt

Results mismatch using released BLEURT-Large-128

Open
#35 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
793
Forks
94
PR merge metrics
No merged PRs in 30d

Description

Hi there,

Recently I'm interested in reimplementing and investigating your research. However, when I directly use your released code and BLEURT-Large-128 checkpoint model, I can't get comparable results with what you present [here](https://github.com/google-research/bleurt/blob/master/checkpoints.md). Here is what I got:

- de-en: 29.15
- fi-en: 30.64
- gu-en: 27.49
- kk-en: 39.14
- lt-en: 34.28
- ru-en: 26.75
- zh-en: 41.98
- avg: 32.78

I first follow your command to get the WMT2019 data and the BLEURT-Large-128 checkpoint. After evaluating the whole dataset file, I collect the prediction scores, split the results and corresponding golden scores by language pairs, and compute Kendall results using scipy.stats.kendalltau as your implementation.

So I'm wondering I've missed any detail. Could you help me? Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.