NVIDIA / NVIDIA/TensorRT

Requesting a new workflow that aids with using engines build against different runtimes

Open
#4,530 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature Request triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Hey, I consume SDKs made by 3rd party vendors that make use of TensorRT and ship with their own models/engines. This means different vendors will be using different versions of TensorRT. For example Nvidia Maxine and Stereolabs ZED.
I have no control over the TensorRT version, nor do I have the ability to re-build the engines against a common version (since they are not mine).
The issue is that since the binary name is the same for all versions of TRT10 (nvinfer_10.dll), I'm unable to load engines from both SDKs into the process at the same time since only one copy of a binary with that name can be loaded at a time.
I understand there is compatibility mode available when building engines, but it doesn't seem like that is used, and I have no way to make the SDK vendors make use of it.

It seems to me that the fact that nvinfer_10.dll breaks binary compatibility rules, that a newer version should be able to be loaded by a process and existing workflows built against an older one should continue to work, is something that should be avoidable.

I would like to suggest TensorRT builds also come with a .lib an .dll pair that are named with their full version, so that we can load multiple copies of TensorRT into a process at the same time, and each SDK will get the version they need. The .lib can likely stay with a common name to make build systems easier to maintain, and just the .dll ends up being different for each TensorRT release.
SDK vendors can be encouraged to link against this .lib instead, which ensure their SDK is always loading their exact required TensorRT version.

Thanks for your consideration

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the TensorRT runtime packaging and Windows DLL naming described in the issue, including how SDK vendors link against the accompanying .lib. Done means proposing and implementing a version-specific .dll/.lib arrangement that allows multiple TensorRT runtime versions to load in one process without breaking existing workflows.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.