microsoft / microsoft/onnxruntime
[Feature Request] Add `use_tf32` flag for TensorRT
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Currently the only way to disable TF 32 from python when using TensortrtExecutionProvider is setting the environment variable `NVIDIA_TF32_OVERRIDE=0` whereas CUDAExecutionProvider has an expliclit `use_tf32` configuration option: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#use_tf32.
I have also found that the environment variable appears to keep it's value once the first session has been created, making it not possible to have one session with it set to `1` and another set to `0`.
This is an environment variable that often needs to be set in environments where precision and reproducibility are critical and often causes unexpected results when upgrading from older GPUs to those with TF32 enabled by default.
### Describe scenario use case
This has came up many times in issues:
- https://github.com/microsoft/onnxruntime/issues/22354
- https://github.com/microsoft/onnxruntime/issues/21457
- https://github.com/microsoft/onnxruntime/issues/19288
- https://github.com/microsoft/onnxruntime/issues/6427
It would make profiling with different settings easier.
Contributor guide
Research direction
Start with the TensorRT execution provider configuration and compare it with the CUDAExecutionProvider's documented use_tf32 option. Trace how NVIDIA_TF32_OVERRIDE is read and retained across sessions, then verify that separate sessions can select TF32 independently and that the existing TensorRT provider behavior remains covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100