'notifyShape' API provides a -1 dimension on TensorRT 10.16.1 when running Qwen2.5-VL-3B on Jetson Thor

Description

When IExecutionContext::enqueueV3() calls IOutputAllocator::notifyShape() for a dynamic-shaped output, the supplied Dims contains a -1 for the dynamic axis even though TensorRT has internally resolved that axis. The same context's getTensorShape() also returns -1 for that axis at the same point, but getTensorStrides() returns strides that are only consistent with a fully-resolved shape. This issue does not occur on TensorRT 10.12 x86, but for the exact same ONNX model it does occur for TensorRT 10.16.1 aarch64 (Jetson Thor). Concretely, on a VLM (Qwen2.5-VL-3B) decoder engine the KV-cache output is reported as:

notifyShape dims = [1, 8, -1, 128]
getTensorShape = [1, 8, -1, 128]
getTensorStrides = [277504, 34688, 128, 1] ⇒ implies dim 2 = 34688/128 = 271
reallocateOutputAsync size > 0

The strides and the positive size from reallocateOutputAsync prove TensorRT internally computed the shape (271 in dim 2). The header for notifyShape documents that it is "Called by TensorRT when the shape of the output tensor is known", but this is not occurring correctly in the callback path.

All inputs had setInputShape called with concrete dims before enqueueV3().

Environment

TensorRT Version: 10.16.1.11 (pinned as TensorRT-10.16.1.11.Linux.aarch64-gnu.cuda-13.2)

NVIDIA GPU: NVIDIA Thor (Jetson Thor), compute capability 11.0, 20 SMs, 125 GiB total memory

NVIDIA Driver Version: 580.00

CUDA Version: 13.0 (runtime/driver) — TRT built against CUDA 13.2 headers

CUDNN Version:

Operating System: Ubuntu 24.04 aarch64 (Jetson Thor)

Relevant Files

Model link:

Steps To Reproduce

1. Build a TRT engine from an ONNX export of a Qwen2.5-VL decoder block exposing dynamic
   past-sequence-length on the KV-cache I/O (past_key_values..{key,value} as input,
   present_key_values..{key,value} as output, dim 2 = past_seq_len, dynamic). Optimization profile sets
   a finite range for that axis. Built with polygraphy.backend.trt.CreateConfig(strongly_typed=True,
   hardware_compatibility_level=SAME_COMPUTE_CAPABILITY) and INT4-AWQ weight quantization.
2. At runtime: setInputShape called with concrete dims for every dynamic input, setInputTensorAddress
   for inputs, setOutputAllocator for dynamic outputs (present_key_values..{key,value}), then
   enqueueV3().
3. Inside notifyShape, the callback fires for present_key_values.0.value with dims=[1, 8, -1, 128] while
   getTensorStrides returns [277504, 34688, 128, 1] (consistent with [1, 8, 271, 128]) and getTensorShape
   returns the same [1, 8, -1, 128].

Commands or scripts:

Have you tried the latest release?: Yes, this is on the latest release

Attach the captured .json and .bin files from TensorRT's API Capture tool if you're on an x86_64 Unix system

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): This model can be run on earlier versions of TensorRT

Start with the IOutputAllocator::notifyShape() callback path invoked by enqueueV3(), then compare its Dims with getTensorShape(), getTensorStrides(), and reallocateOutputAsync(). Reproduce the Qwen2.5-VL-3B decoder case on TensorRT 10.16.1 aarch64 and compare it with TensorRT 10.12 x86. Done means the dynamic KV-cache output shape reported to notifyShape and getTensorShape agrees with the resolved strides and allocation size.