microsoft / microsoft/onnxruntime

[Feature Request] Add `use_tf32` flag for TensorRT

Open
#26,029 3 comments 0 reactions 0 assignees View on GitHub
ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Currently the only way to disable TF 32 from python when using TensortrtExecutionProvider is setting the environment variable `NVIDIA_TF32_OVERRIDE=0` whereas CUDAExecutionProvider has an expliclit `use_tf32` configuration option: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#use_tf32.

I have also found that the environment variable appears to keep it's value once the first session has been created, making it not possible to have one session with it set to `1` and another set to `0`.

This is an environment variable that often needs to be set in environments where precision and reproducibility are critical and often causes unexpected results when upgrading from older GPUs to those with TF32 enabled by default.

### Describe scenario use case

This has came up many times in issues:

- https://github.com/microsoft/onnxruntime/issues/22354
- https://github.com/microsoft/onnxruntime/issues/21457
- https://github.com/microsoft/onnxruntime/issues/19288
- https://github.com/microsoft/onnxruntime/issues/6427

It would make profiling with different settings easier.

Contributor guide

Open the contributing guide

Research direction

Start with the TensorRT execution provider configuration and compare it with the CUDAExecutionProvider's documented use_tf32 option. Trace how NVIDIA_TF32_OVERRIDE is read and retained across sessions, then verify that separate sessions can select TF32 independently and that the existing TensorRT provider behavior remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.