Weird behavior on V100 32GB

Hi.

I have been doing some benchmarks on nvidia V100 32GB gpu.

First, I benchmarked Llama2-7B-chat using huggingface transformers and CTranslate2. I saw reduced latency when using ct2 ( 12 seconds vs 7.5 seconds respectively)

However, when I tried the 13B version, I didn't see any improvement in latency at all (18 seconds vs 18 seconds) although there's a little bit of reduction of vRAM.

Why is this happening? Did I do something wrong?

This is the code that I am using

input = llama2_chat_prompt_template.format(transcript=transcript)

start = time.time()

tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(input))
results = generator.generate_batch([tokens], max_length=512, include_prompt_in_result=False)
output = tokenizer.decode(results[0].sequences_ids[0])

end = time.time()

t = end-start

print(f"GPU:\tV100\nTime(s):\t{t}\nResult: {output}")

No repository file, test, or entry point is named. Start by reproducing the supplied benchmark on the V100 with the 7B and 13B models, then compare the Hugging Face Transformers and CTranslate2 runs; done means explaining or isolating the latency difference with repeatable measurements.