Other differences in the beam search implementation?

Just to make sure that issue won't be missed, will duplicate my response here:

I faced the same issue. However, when read this response, I thought that if I didn't use any special generation parameters (like no_repeat_ngram_size), I would get the same result. Unfortunately, seems that there are other differences in beam search implementation - or do I miss something?

To reproduce:

package versions: transformers==4.34.0, ctranslate2==3.20.0 (as used here)

1. convert model

ct2-transformers-converter --model "google/flan-t5-base" --output_dir "ct2-t5-base"

2. code snippet:

import torch

from transformers import T5ForConditionalGeneration, AutoTokenizer
import ctranslate2


device = torch.device("cuda")

model_name = "google/flan-t5-base"
hf_model = T5ForConditionalGeneration.from_pretrained(model_name).eval().to(device)
tokenizer = AutoTokenizer.from_pretrained(model_name)


fast_model =  ctranslate2.Translator("ct2-t5-base", device="cuda")


text = "translate English to German: physician assistants are medical providers who are licensed to diagnose and treat illness and disease and to prescribe medication"


def get_out(inp, model):
    inputs = tokenizer(inp, return_tensors="pt")
    ids = model.generate(**inputs.to(device),
                         num_beams=3,
                         min_length=0,
                         max_length=1024,
                         )
    return tokenizer.batch_decode(ids, skip_special_tokens=True)[0]


def get_out_fast(inp, model):
    source = tokenizer.encode(inp)
    source = tokenizer.convert_ids_to_tokens(source)
    results = model.translate_batch([source],
                                    beam_size=3,
                                    min_decoding_length=0,
                                    max_decoding_length=1024)
    target = results[0].hypotheses[0]
    return tokenizer.decode(tokenizer.convert_tokens_to_ids(target), skip_special_tokens=True)


res_vanilla = get_out(text, hf_model)
res_fast = get_out_fast(text, fast_model)


print("Vanilla output:", res_vanilla)
print("Ctranslate output:", res_fast)

Output:

Vanilla output: physician assistants sind medical providers, die zu Diagnose und Behandlung von Krankheiten und Krankheiten und zu Verknüpfen von Medikamenten zu ermitteln.
Ctranslate output: physician assistants sind medical providers, die zu Diagnose und Behandlung von Krankheiten und Krankheiten und zu Verknüpfen von Medikamenten zu kaufen sind.

Start by running the provided google/flan-t5-base conversion and Python comparison using beam_size/num_beams=3. Compare the Transformers and CTranslate2 outputs and investigate the beam-search behavior responsible for the difference. Done means identifying and documenting the implementation difference, with a regression test or clear reproduction confirming the result.