microsoft / microsoft/onnxruntime

Tensorrt cache is not being re-used with dynamic dimensions

Open
#16,889 4 comments 0 reactions 1 assignee Claimed by @chilo-ms View on GitHub
ep:TensorRT
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Using onnxruntime with TensorrtExecutionProvider rebuilds the cache engine when you pass the trt_profile_min_shapes, trt_profile_opt_shapes, trt_profile_max_shapes.
If I first build the cache with dynamic shapes and in the next inference do not define the trt_profile shapes, then the cache is used.

I am using:
`onnxruntime-gpu==1.15.0`

### To reproduce

Run twice the example:
```
import onnxruntime as ort

ort.set_default_logger_severity(0) # Turn on verbose mode for ORT TRT
sess_options = ort.SessionOptions()

trt_ep_options = {
"trt_fp16_enable": True,
"trt_engine_cache_enable": True,
"trt_profile_min_shapes": "sample:2x4x64x64,encoder_hidden_states:2x77x768",
"trt_profile_max_shapes": "sample:32x4x64x64,encoder_hidden_states:32x77x768",
"trt_profile_opt_shapes": "sample:2x4x64x64,encoder_hidden_states:2x77x768",
}

sess = ort.InferenceSession(
"my_model.onnx",
providers=[
("TensorrtExecutionProvider", trt_ep_options),
"CUDAExecutionProvider",
],
)

batch_size = 1
unet_dim = 4
max_text_len = 77
embed_dim = 768
latent_height = 64
latent_width = 64

args = {
"sample": np.zeros(
(2 * batch_size, unet_dim, latent_height, latent_width), dtype=np.float32
),
"timestep": np.ones((1,), dtype=np.float32),
"encoder_hidden_states": np.zeros(
(2 * batch_size, max_text_len, embed_dim),
dtype=np.float32,
),
}
sess.run(None, args)
```

### Urgency

Low.

### Platform

Linux

### OS Version

Ubuntu 20.04

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

onnxruntime-gpu==1.15.0

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

TensorRT

### Execution Provider Library Version

docker image: nvcr.io/nvidia/tensorrt:22.12-py3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.