[Usage]: Does inflight batching support batched tensors as input?

System Info

System Information:

• OS: Ubuntu 24.04
• CUDA version: 13
• TensorRT-LLM version: 25.08 NGC container

How would you like to use TensorRT-LLM

I want to run inference on Whisper BLS from your examples. I don't know how to integrate it with TensorRT-LLM or optimize it for my use case.

Specific questions:

• Model: Tensorrt LLM, Whisper BLS (Your example)
• Use case : Inference
• Expected throughput/latency requirements: Maximum throughput, latency not important

I searched through the whole documentation, but I was not able to find an answer.
I am currently implementing a variant for your Whisper BLS example which uses the tensorrt_llm backend.
While implementing batching for it I noticed that tensorrt_llm does not use inflight_batching for
tensors of shape (batch_size, 3000, 128) - but does so for (3000, 128).

So is this a mistake on my side, should I handle this differently or not use inflight batching at all? I thought it would be neat if it was possible to accumulate multiple batched tensors into a bigger batch.

I am currently not concerned with latency, just throughput my thinking was batching requests before sending them into tensorrt_llm would have benefits.

What I saw was that tensorrt_llm expects tensors to be packed in order to use inflight batching. At the moment all the inputs have shape (batch_size, ...). I thought about using them packed but I found that there is not a input length parameter for all necessary inputs to make the packed tensors work.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start with the Whisper BLS example and its tensorrt_llm backend path, then trace how inputs shaped (batch_size, 3000, 128) and (3000, 128) are passed to inflight batching. Review the documentation and examples for packed inputs and input-length parameters. Done means the supported batching behavior and any required input metadata are clearly established for this use case.