[Usage]: Mistral 3.1 Torch Backend

System Info

I am getting no latency improvement when using Mistral 3.1 Small using TensorRT torch backend compared to using HuggingFace. They both take about 21-22 seconds when inferring upon a request containing 5 images. Is this expected?

This is the command I am running.:

trtllm-serve ./model3 --tp_size 8 --max_num_tokens 10065 --max_batch_size 1 --extra_llm_api_options ./extra-llm-api-config.yml --host 0.0.0.0 --port 8000

My config.yml only contains:

kv_cache_config: enable_block_reuse: false
The model needs to be able to take in 5 images at a time hence the max_num_tokens. Is there any way to improve upon this? Or is there any plans for trt engine support?

Start with the trtllm-serve command, model3, and extra-llm-api-config.yml configuration shown in the report, then compare the TensorRT torch backend and HuggingFace runs under the stated five-image workload. Done means determining whether the similar 21–22 second latency is expected, identifying a supported optimization path, or documenting the status of TensorRT engine support.