microsoft / microsoft/onnxruntime
[Feature Request] Is there any option to profile op on TensorRT EP?
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
https://onnxruntime.ai/docs/performance/tune-performance/profiling-tools.html
☝️ as document said, I thought `--enable_cuda_profiling` might be helpful profiling operators onnx model on using tensor rt ep. But it couldn't help. it only shows total tensorrt engine latency
Is there any way to profile each op latency on tensor rt ep?
### Describe scenario use case
1. build onnxruntime tensorrt docker using --enable_cuda_profiling
https://github.com/microsoft/onnxruntime/blob/main/dockerfiles/Dockerfile.tensorrt
2. inference model using TensorrtExecutionProvider
providers = ['CUDAExecutionProvider']
providers.insert(0, ('TensorrtExecutionProvider'))
3. op profiling result

Not showing each op latency result.
Contributor guide
Research direction
Start with the profiling tools documentation and dockerfiles/Dockerfile.tensorrt, then reproduce the reported result using the shown TensorrtExecutionProvider configuration. Compare the output with the expected per-operator latency and determine the TensorRT profiling entry point needed to expose it. Done means a defined way to obtain per-op latency, with documentation or validation of the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, docker, python
- Domain
- machine-learning, observability, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100