microsoft / microsoft/onnxruntime

[Feature Request] Is there any option to profile op on TensorRT EP?

Open
#18,803 0 comments 0 reactions 0 assignees View on GitHub
ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

https://onnxruntime.ai/docs/performance/tune-performance/profiling-tools.html

☝️ as document said, I thought `--enable_cuda_profiling` might be helpful profiling operators onnx model on using tensor rt ep. But it couldn't help. it only shows total tensorrt engine latency

Is there any way to profile each op latency on tensor rt ep?

### Describe scenario use case

1. build onnxruntime tensorrt docker using --enable_cuda_profiling
https://github.com/microsoft/onnxruntime/blob/main/dockerfiles/Dockerfile.tensorrt

2. inference model using TensorrtExecutionProvider
providers = ['CUDAExecutionProvider']
providers.insert(0, ('TensorrtExecutionProvider'))

3. op profiling result
![8b64c10c-069a-4f18-beae-a00093b2267f](https://github.com/microsoft/onnxruntime/assets/9348174/c8b9b390-f334-46fc-9177-ce04f794a7e1)

Not showing each op latency result.

Contributor guide

Open the contributing guide

Research direction

Start with the profiling tools documentation and dockerfiles/Dockerfile.tensorrt, then reproduce the reported result using the shown TensorrtExecutionProvider configuration. Compare the output with the expected per-operator latency and determine the TensorRT profiling entry point needed to expose it. Done means a defined way to obtain per-op latency, with documentation or validation of the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, python
Domain
machine-learning, observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.