microsoft / microsoft/onnxruntime

[Feature Request] Add ability to set memory pool limit for tactic search in TensorRT EP

Open
#24,502 1 comment 0 reactions 0 assignees View on GitHub
ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Engine creation by TensorRT uses the internal flag nvinfer1::MemoryPoolType::kTACTIC_DRAM to set a limit for tactic search, without this memory during tactic selection (i.e. engine creation) is not set and significantly higher spikes of memory is observed than set by workspace limit (nvinfer1::MemoryPoolType::kWORKSPACE)
I want to add the ability to set this variable in the TensorRT EP via a EP session flag.

Contributor guide

Open the contributing guide

Research direction

Start with TensorRT EP session-flag handling and the engine-creation path where MemoryPoolType::kTACTIC_DRAM and kWORKSPACE are used. Define completion as allowing the tactic-search memory limit to be set independently through an EP session flag, then verify that the configured limit affects engine creation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.