TensorRT-Edge-LLM Whisper Support

Description

We are deploying OpenAI Whisper (encoder-decoder speech recognition model) on NVIDIA Jetson Thor (SM 110). We would like to request Whisper model family support in TensorRT-Edge-LLM.

Current Situation

We attempted to use TensorRT-LLM (v1.2.0rc8 and v1.3.0rc2) for Whisper on Jetson Thor, but discovered:

FMHA kernels crash on SM 110 - Both BertAttentionPlugin and GPTAttentionPlugin fail during engine build with a segfault in FmhaDispatcher constructor, as the Flash Multi-Head Attention kernels are not compiled for SM 110 (compute capability 11.0)

Stack trace (abbreviated):

libtensorrt_llm.so(
  _ZN12tensorrt_llm3_v17kernels14FmhaDispatcherC2ENS1_20MHARunnerFixedParamsE
)
libnvinfer_plugin_tensorrt_llm.so(
  _ZN12tensorrt_llm7plugins19BertAttentionPlugin10initializeEv
)
*** Process received signal: Aborted (6) ***

TensorRT-Edge-LLM supports Jetson Thor (SM 110) — confirmed via the issue template — but currently only supports LLMs and VLMs, not encoder-decoder speech models.

As a workaround, we use ONNX Runtime with TensorRT Execution Provider on Jetson Thor:

• Encoder: TensorRT EP (~5-10ms)
• Decoder: CUDA EP (~100-150ms) — TensorRT EP struggles with dynamic shapes

This works but is suboptimal. Native TensorRT-Edge-LLM support with proper KV-cache management and FMHA kernels for SM 110 would significantly improve decoder performance.

Requests

1. Whisper model family (small, medium, large-v3, distil variants) to the supported models list
2. Support for HuggingFace fine-tuned Whisper checkpoints (not just OpenAI originals)
3. Encoder-decoder architecture with cross-attention support
4. FMHA kernels work on SM 110 for both self-attention and cross-attention

Environment

• Hardware: NVIDIA Jetson Thor
• GPU Compute Capability: SM 110 (11.0)
• Driver: 580.00
• CUDA: 13.0
• JetPack: 7.x
• TensorRT: 10.13.x

Related

• NVIDIA Developer Forums: TensorRT-LLM not available on Jetson
• NVIDIA-AI-IOT/whisper_trt — TensorRT Whisper for Jetson Orin, but only supports original OpenAI models
• TensorRT-LLM Whisper example — Works on x86 GPUs but crashes on Jetson Thor due to FMHA

Start by comparing the TensorRT-LLM Whisper example at examples/models/core/whisper with NVIDIA-AI-IOT/whisper_trt and the repository's supported-models structure. Trace how encoder-decoder cross-attention, KV-cache handling, and FMHA kernels are represented, then define completion as Whisper checkpoints running natively on Jetson Thor with working SM 110 attention support.