Can't build whisper engines with past two releases

System Info

• CPU architecture: x86_64
• CPU/Host memory size: 512GB
• GPU properties
  • GPU name: 2xNVIDIA H100
• Libraries
  • TensorRT-LLM branch or tag: v0.16.0dev2024111900 or higher
  • Versions of TensorRT, Modelopt, CUDA, cuBLAS, etc. used: the ones in the requirements

Who can help?

@byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

trtllm-build  --checkpoint_dir trt/whisper_large-v2_weights_int8/encoder \
              --output_dir trt/whisper_large-v2_int8_2/encoder \
              --kv_cache_type paged \
              --moe_plugin disable \
              --enable_xqa enable \
              --max_batch_size 16 \
              --gemm_plugin disable \
              --bert_attention_plugin float16 \
              --max_input_len 3000 \
              --max_seq_len 3000 \
              --log_level debug \
              --enable_debug_output

Expected behavior

Engine builds successfully and fast

actual behavior

Whisper Large-V2 encoder hangs when building the engine
this is the last thing printed before it hangs: [11/27/2024-23:00:09] [TRT] [V] Disabling unused tactic source: JIT_CONVOLUTIONS
I tried waiting for around an hour but it still didn't complete and nvidia-smi memory usage for the process was constant

Tiny model builds fine but it's much slower than older versions

additional notes

None