Reported results cant be achieved in Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

@patrickvonplaten
I have been trying to achieve a bleu score of 31.7 (as reported in the blog and paper for WMT en->de evaluation) using hugging-face model **google/bert2bert_L-24_wmt_en_de** but I could only achieve 23.77 on newstest2014 test set . I have kept the beam search config as mentioned in the paper that is num_beams = 4 and length penalty = 0.6 , fixed the max length = 128 as was done during training in paper. I have also used the bleu script mentioned in the footnotes .

Can you please tell what could be missing in this whole process and how can I achieve the similar scores?

Start by comparing the blog and paper's WMT en-de setup with the google/bert2bert_L-24_wmt_en_de evaluation, including beam search, length penalty, max length, BLEU script, and newstest2014. Done means identifying the missing evaluation detail or reproducing the reported 31.7 BLEU score.

Written by the indexing model from the issue text.