microsoft / microsoft/onnxruntime

[Performance] NV TRT RTX provider performance slower than TensorRT on RTX 4000 Ada

Open
#25,442 9 comments 0 reactions 1 assignee Assigned to @hariharans29 View on GitHub
ep:TensorRT performance
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

When running inference using the NV TRT RTX execution provider in ONNX Runtime on an RTX 4000 Ada GPU, I observed that it is slower than the TensorRT execution provider for the same model.

**Performance observation**
- TensorRT provider inference time: 40 seconds
- NV TRT RTX provider inference time: 60 seconds

**Expected behavior**
NV TRT RTX execution provider should perform as well as or better than TensorRT provider on the same hardware and model.

Please let me know if further information, logs, or model files are needed to diagnose this issue.

### To reproduce

1. Set up ONNX Runtime with NV TRT RTX and TensorRT execution providers.
2. Run the same ONNX model on an NVIDIA RTX 4000 Ada GPU with identical input shapes and batch size.
3. Observe and compare the inference times.
4. (Model name, input shape, and exact environment details to be filled in.)

### Urgency

_No response_

### Platform

Linux

### OS Version

sles

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

a0c42367b7f74db0f529c36d7bcfc0a8c83b9f3d

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

TensorRT, Other / Unknown

### Execution Provider Library Version

_No response_

### Model File

_No response_

### Is this a quantized model?

No

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.