Performance Regression in Whisper models when timestamp generation is enabled

Hello
Several reports mention that WER improves greatly when adding <|notimestamps|> to the initial prompt in whisper decoding aka disabling timestamps generation, I tested this using This and This. You can check https://github.com/mobiusml/faster-whisper/pull/18#issuecomment-2196788100 for an example of decoding difference using the same encoder output
There are several other reports including but not limited to:
https://github.com/SYSTRAN/faster-whisper/discussions/1010
https://github.com/SYSTRAN/faster-whisper/issues/985

Also generation with timestamps has a lower toks/s and the slowdown increases when increasing the batch size

on the side, we have several PRs waiting for @trungkienbkhn review but he seems to be out of office, it'd be great if one of his colleagues has any information when he might return

Start with benchmark/wer_benchmark.py and benchmark/evaluate_yt_commons.py, then compare Whisper decoding with and without timestamp generation. Check the linked reports and decoding example for differences in WER and tokens per second, especially as batch size increases. The issue does not name a CTranslate2 source file or test, and its acceptance criteria remain unresolved.