microsoft / microsoft/onnxruntime
Tensorrt cache is not being re-used with dynamic dimensions
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
Using onnxruntime with TensorrtExecutionProvider rebuilds the cache engine when you pass the trt_profile_min_shapes, trt_profile_opt_shapes, trt_profile_max_shapes.
If I first build the cache with dynamic shapes and in the next inference do not define the trt_profile shapes, then the cache is used.
I am using:
`onnxruntime-gpu==1.15.0`
### To reproduce
Run twice the example:
```
import onnxruntime as ort
ort.set_default_logger_severity(0) # Turn on verbose mode for ORT TRT
sess_options = ort.SessionOptions()
trt_ep_options = {
"trt_fp16_enable": True,
"trt_engine_cache_enable": True,
"trt_profile_min_shapes": "sample:2x4x64x64,encoder_hidden_states:2x77x768",
"trt_profile_max_shapes": "sample:32x4x64x64,encoder_hidden_states:32x77x768",
"trt_profile_opt_shapes": "sample:2x4x64x64,encoder_hidden_states:2x77x768",
}
sess = ort.InferenceSession(
"my_model.onnx",
providers=[
("TensorrtExecutionProvider", trt_ep_options),
"CUDAExecutionProvider",
],
)
batch_size = 1
unet_dim = 4
max_text_len = 77
embed_dim = 768
latent_height = 64
latent_width = 64
args = {
"sample": np.zeros(
(2 * batch_size, unet_dim, latent_height, latent_width), dtype=np.float32
),
"timestep": np.ones((1,), dtype=np.float32),
"encoder_hidden_states": np.zeros(
(2 * batch_size, max_text_len, embed_dim),
dtype=np.float32,
),
}
sess.run(None, args)
```
### Urgency
Low.
### Platform
Linux
### OS Version
Ubuntu 20.04
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
onnxruntime-gpu==1.15.0
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
TensorRT
### Execution Provider Library Version
docker image: nvcr.io/nvidia/tensorrt:22.12-py3
Contributor guide
Assessment
This issue has not been assessed yet.