microsoft / microsoft/onnxruntime
[Feature Request] Need to have explicit, per-session control of TF32 use
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Currently, ORT enables TF32 by default.
The only way to override that is from the environment variable.
Unfortunately, that's not enough for the use case of Triton having 2 ORT components, when one wants to disable TF32 and the other wants to use it.
PyTorch, for example, have explicit settings in its runtime that can be toggled at any time to achieve desired effect.
Would be nice to have something like that in ORT.
### Describe scenario use case
Use case of Triton, for example, having 2 ORT backend components, when one wants to disable TF32 and the other is not.
Contributor guide
Research direction
Start by reviewing ONNX Runtime's existing TF32 environment-variable behavior and how session settings are represented. Compare the requested per-session control with the runtime toggles described for PyTorch and the two-component Triton scenario. Done means each ORT session can independently enable or disable TF32 without changing process-wide behavior.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100