NVIDIA / NVIDIA/TensorRT

Trt-engine is 2x slower than the onnx model

Open
#4,519 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:ONNX Module:Performance triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

I tried to inference the sam2-decoder with trt-engine, but, as you can see from the screenshots below, it is much slower than the inference with onnx model.
TRT
Image
ORT
Image

For better investigating, the onnx model is in attachment.

Any clue would be appreciated, thanks in advance!

Environment

TensorRT Version: 10.3.0

NVIDIA GPU: Jetson Orin 64GB

NVIDIA Driver Version: 540.4.0

CUDA Version: 12.6

CUDNN Version: 9.3.0

Operating System:

Python Version (if applicable): 3.10.12

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:
Model for Debugging

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names only the attached ONNX model and the sam2-decoder; no repository file, test, or runnable command is provided. Start by reproducing the TensorRT and ONNXRuntime comparison in the listed environment, then document the cause and verify that TensorRT performance is comparable under the same conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.