microsoft / microsoft/onnxruntime
TensorRT EP engine cache reuses engine for same graph name with different static input shape
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
When `TensorrtExecutionProvider` engine caching is enabled, two different ONNX models with the same graph name and the same structure but different static input shapes can collide in the TensorRT engine cache.
In the minimal reproducer below:
- Model A is `Add(x, y)` with static input shape `[3, 3]` and graph name `main_graph`.
- Model B is `Add(x, y)` with static input shape `[4, 4]` and the same graph name `main_graph`.
- Model B passes when run alone.
- Running A first and then B with the same `trt_engine_cache_path` makes B fail with TensorRT `setInputShape()` static dimension mismatch.
- Disabling engine cache, changing the graph names, or using different `trt_engine_cache_prefix` values makes both models pass.
This looks like the default TensorRT EP engine cache key/prefix does not distinguish static input shapes when graph names collide, so the engine generated for the 3x3 model is reused for the 4x4 model.
I found #22179, but this case does not use `trt_dump_ep_context_model` or weight-stripped engines. It reproduces with a single Add node and `trt_engine_cache_enable=True`.
### To reproduce
```python
import json
import tempfile
import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper
def make_add_model(n: int, graph_name: str = "main_graph") -> onnx.ModelProto:
x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [n, n])
y = helper.make_tensor_value_info("y", TensorProto.FLOAT, [n, n])
z = helper.make_tensor_value_info("z", TensorProto.FLOAT, [n, n])
graph = helper.make_graph(
[helper.make_node("Add", ["x", "y"], ["z"])],
graph_name,
[x, y],
[z],
)
model = helper.make_model(graph, opset_imports=[helper.make_operatorsetid("", 17)])
model.ir_version = 8
return model
def run_one(label: str, n: int, cache_dir: str, *, cache=True, graph_name="main_graph", prefix=None):
opts = {}
if cache:
opts = {"trt_engine_cache_enable": True, "trt_engine_cache_path": cache_dir}
if prefix is not None:
opts["trt_engine_cache_prefix"] = prefix
providers = [
("TensorrtExecutionProvider", opts),
"CUDAExecutionProvider",
"CPUExecutionProvider",
]
so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
so.log_severity_level = 4
sess = ort.InferenceSession(
make_add_model(n, graph_name).SerializeToString(), so, providers=providers
)
feed = {
"x": np.ones((n, n), dtype=np.float32),
"y": np.ones((n, n), dtype=np.float32),
}
out = sess.run(None, feed)
return [list(np.asarray(v).shape) for v in out]
def run_sequence(name, *, cache=True, graph_names=("main_graph", "main_graph"), prefixes=(None, None)):
cache_dir = tempfile.mkdtemp(prefix=f"ort_trt_cache_{name}_")
result = []
for label, n, graph_name, prefix in [
("A_3x3", 3, graph_names[0], prefixes[0]),
("B_4x4", 4, graph_names[1], prefixes[1]),
]:
try:
result.append(
{
"label": label,
"status": "PASS",
"shapes": run_one(
label,
n,
cache_dir,
cache=cache,
graph_name=graph_name,
prefix=prefix,
),
}
)
except Exception as exc:
result.append(
{
"label": label,
"status": "ERROR",
"type": type(exc).__name__,
"detail": str(exc),
}
)
break
print(name, json.dumps(result, indent=2))
return result
print("onnxruntime", ort.__version__)
print("available_providers", ort.get_available_providers())
# Unexpected: B fails because it reuses the TensorRT engine cached for A.
run_sequence("same_graph_name_cache_enabled")
# Controls: all pass.
run_sequence("cache_disabled", cache=False)
run_sequence("different_graph_names", graph_names=("graph_A", "graph_B"))
run_sequence("different_cache_prefixes", prefixes=("A", "B"))
```
### Expected behavior
Both models should run successfully. In particular, the 4x4 Add model should not reuse an incompatible engine built for the 3x3 model.
### Actual behavior
`same_graph_name_cache_enabled` fails on the second model:
```text
onnxruntime 1.17.1
available_providers ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']
same_graph_name_cache_enabled [
{
"label": "A_3x3",
"status": "PASS",
"shapes": [[3, 3]]
},
{
"label": "B_4x4",
"status": "ERROR",
"type": "RuntimeException",
"detail": "[ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : ... TensorRT EP failed to call nvinfer1::IExecutionContext::setInputShape() for input 'x'"
}
]
```
The TensorRT log contains:
```text
Error Code 3: API Usage Error
Parameter check failed at: runtime/api/executionContext.cpp::setInputShape::2264
condition: engineDims.d[i] == dims.d[i]
Static dimension mismatch while setting input shape.
```
The control cases pass:
```text
cache_disabled: A_3x3 PASS, B_4x4 PASS
different_graph_names: A_3x3 PASS, B_4x4 PASS
different_cache_prefixes: A_3x3 PASS, B_4x4 PASS
```
I repeated the default-cache case three times on the same setup; it failed 3/3. The controls passed.
### Environment
```text
date: 2026-08-25T14:53:20+08:00
host: autodl-container-21104cb00b-3b1b3d4b
python: Python 3.10.20
onnxruntime/numpy/onnx:
onnxruntime 1.17.1
providers ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']
numpy 1.26.4
onnx 1.17.0
nvidia-smi:
NVIDIA GeForce RTX 3080 Ti, 580.105.08, 12288 MiB
nvcc:
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
TensorRT path: /root/TensorRT-8.6.1.6
lrwxrwxrwx 1 root root 19 Apr 27 2023 /root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib/libnvinfer.so -> libnvinfer.so.8.6.1
lrwxrwxrwx 1 root root 19 Apr 27 2023 /root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib/libnvinfer.so.8 -> libnvinfer.so.8.6.1
-rwxr-xr-x 1 root root 235369632 Apr 27 2023 /root/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/lib/libnvinfer.so.8.6.1
```
### Possible workaround
Use unique graph names or set distinct `trt_engine_cache_prefix` values for models with different static input shapes, or disable TensorRT engine caching.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied Python reproducer with the TensorrtExecutionProvider and its cache options, confirming that the same-graph-name sequence fails while the control cases pass. Then inspect the TensorRT EP engine-cache key or prefix handling; done means models with identical graph names but different static input shapes no longer reuse an incompatible engine and the reproducer passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100