microsoft / microsoft/onnxruntime

[Feature Request] TensorRT custom engine Plans

Open
#13,559 12 comments 0 reactions 1 assignee Claimed by @stevenlix View on GitHub
ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

It would be great to have the option to provide pre-optimised TensorRT engine plans to ORT.

### Describe scenario use case

Using TensorRT in standalone, e.g. trtexec, allows for much more precise control over engine compilation.

Currently, only a very small amount of TRT features is accessible through the ORT API, but most importantly, there is no way to ship precompiled engines with an application; Engine compilations need to be done on the user machine, resulting in a bad user experience.
Also, how (TRT) caching is handled internally is not very transparent, in my opinion, and might lead to unnecessary engine recompilations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.