🐛 [Bug] Encountered bug when using Torch-TensorRT

Bug Description

When compiling a pytorch model with torch-tensorrt, using different compiling methods I get different errors:

using this code:

head_jit = torch.jit.trace(om.head, idd)
trt_head2 = torch_tensorrt.compile(
head_jit,
ir="ts",
inputs=idd,
debug=True)
torch.jit.save(trt_head2, "trt_head_ts.ts")

I get this error:

WARNING: [Torch-TensorRT] - For input target_labels, found user specified input dtype as Float. The compiler is going to use the user setting Float
WARNING: [Torch-TensorRT] - For input train_ltrb, found user specified input dtype as Float. The compiler is going to use the user setting Float
Traceback (most recent call last):
File "/pytracking-master-dir/band4.py", line 47, in
trt_head2 = torch_tensorrt.compile(
File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 185, in compile
compiled_ts_module: torch.jit.ScriptModule = torchscript_compile(
File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compiler.py", line 151, in compile
compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/partitioning/shape_analysis.cpp:311] Unable to process subgraph input type of at::kLong/at::kDouble, try to compile model with truncate_long_and_double enabled

using this code:

trt_head2 = torch_tensorrt.compile(
om.head,
ir="ts",
inputs=idd,
debug=True)
torch.jit.save(trt_head2, "trt_head_ts.ts")

I get this error:

Traceback (most recent call last):
File "/pytracking-master-dir/band4.py", line 47, in
trt_head2 = torch_tensorrt.compile(
File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 183, in compile
ts_mod = torch.jit.script(module)
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 1324, in script
return torch.jit._recursive.create_script_module(
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 559, in create_script_module
return create_script_module_impl(nn_module, concrete_type, stubs_fn)
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 632, in create_script_module_impl
script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 639, in _construct
init_fn(script_module)
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 608, in init_fn
scripted = create_script_module_impl(
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 632, in create_script_module_impl
script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 639, in _construct
init_fn(script_module)
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 608, in init_fn
scripted = create_script_module_impl(
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 636, in create_script_module_impl
create_methods_and_properties_from_stubs(
File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 469, in create_methods_and_properties_from_stubs
concrete_type._create_methods_and_properties(
RuntimeError: Unsupported value kind: Tensor

using this code:

head_jit = torch.jit.trace(om.head, idd)
trt_head2 = torch_tensorrt.compile(
head_jit,
ir="ts",
inputs=idd,
debug=True,
truncate_long_and_double=True)
torch.jit.save(trt_head2, "trt_head_ts.ts")

I get this error:

WARNING: [Torch-TensorRT] - For input target_labels, found user specified input dtype as Float. The compiler is going to use the user setting Float
WARNING: [Torch-TensorRT] - For input train_ltrb, found user specified input dtype as Float. The compiler is going to use the user setting Float
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Sum converter disregards dtype
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Sum converter disregards dtype
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Unused Input: input_1
WARNING: [Torch-TensorRT TorchScript Conversion Context] - [RemoveDeadLayers] Input Tensor input_1 is unused or used only at compile-time, but is not being removed.
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [elementWiseLayer.cpp::validate::61] Error Code 4: Internal Error ((Unnamed Layer* 13) [ElementWise]: operation SUM requires inputs with activation or Int32 type.)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [elementWiseLayer.cpp::validate::61] Error Code 4: Internal Error ((Unnamed Layer* 21) [ElementWise]: operation SUM requires inputs with activation or Int32 type.)
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 31) [Slice]: output shape can not be computed)
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 31) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%32 : Tensor = aten::sub(%y_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%39 : Tensor = aten::add(%38, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%32 : Tensor = aten::sub(%y_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%39 : Tensor = aten::add(%38, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%32 : Tensor = aten::sub(%y_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%39 : Tensor = aten::add(%38, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed)
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 44) [Slice]: output shape can not be computed)
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 44) [Slice]: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%42 : Tensor = aten::sub(%x_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%46 : Tensor = aten::add(%45, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%42 : Tensor = aten::sub(%x_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%46 : Tensor = aten::add(%45, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%42 : Tensor = aten::sub(%x_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%46 : Tensor = aten::add(%45, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%x_embed.3 : Tensor = aten::div(%42, %46), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%x_embed.3 : Tensor = aten::div(%42, %46), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed)
Traceback (most recent call last):
File "/pytracking-master-dir/band4.py", line 47, in
trt_head2 = torch_tensorrt.compile(
File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 185, in compile
compiled_ts_module: torch.jit.ScriptModule = torchscript_compile(
File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compiler.py", line 151, in compile
compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: [Error thrown at core/util/trt_util.cpp:165] pos >= (-d.nbDims - 1) && pos <= d.nbDims ASSERT FAILED at core/util/trt_util.cpp:165, consider filing a bug: https://www.github.com/NVIDIA/Torch-TensorRT/issues
ERROR: Index to unsqueeze is out of bounds. Expected value in range [0, -1], but got -1

If anybody has any experience, please let me know

I you need more info or the full model, please tell me

THANK YOU

Reproduce the three compilation variants from the issue and inspect the referenced paths in torch_tensorrt/_compile.py, ts/_compiler.py, core/partitioning/shape_analysis.cpp, and ltr/models/transformer/position_encoding.py. Trace the dtype, unsupported-value, and shape-analysis failures to determine whether they share a compiler cause. Done means identifying the root cause and confirming a fix with a regression case for the reported model behavior.