microsoft / microsoft/onnxruntime

[Feature Request] Support 'trt_bf16_enable' in TensorRT EP on GPU

Open
#24,765 1 comment 0 reactions 0 assignees View on GitHub
ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I'm using ONNX Runtime with TensorRT Execution Provider on GPU. I noticed that 'trt_fp16_enable' is supported and works well, but there is no documentation regarding 'trt_bf16_enable'.
- Is there any plan to support BF16 via 'trt_bf16_enable' in the near future?
- Are there any known workarounds or experimental builds for BF16 support on GPU using TensorRT EP?
- If it's not yet supported, is it on the roadmap?

### Describe scenario use case

We are running SLM model on A100 with TensorRT EP, and would like to use BF16 for better performance compared to FP32, while keeping accuracy close.

Contributor guide

Open the contributing guide

Research direction

The request concerns the TensorRT Execution Provider and the trt_bf16_enable option, but names no implementation files or tests. Start by checking the provider's existing trt_fp16_enable support and documentation, then determine the scope of BF16 support. Done would require a decided support path with corresponding implementation and validation details.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.