microsoft / microsoft/onnxruntime

[Feature Request] Need to have explicit, per-session control of TF32 use

Open
#15,407 1 comment 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Currently, ORT enables TF32 by default.
The only way to override that is from the environment variable.
Unfortunately, that's not enough for the use case of Triton having 2 ORT components, when one wants to disable TF32 and the other wants to use it.
PyTorch, for example, have explicit settings in its runtime that can be toggled at any time to achieve desired effect.
Would be nice to have something like that in ORT.

### Describe scenario use case

Use case of Triton, for example, having 2 ORT backend components, when one wants to disable TF32 and the other is not.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing ONNX Runtime's existing TF32 environment-variable behavior and how session settings are represented. Compare the requested per-session control with the runtime toggles described for PyTorch and the two-component Triton scenario. Done means each ORT session can independently enable or disable TF32 without changing process-wide behavior.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.