microsoft / microsoft/onnxruntime
[Feature Request] Support 'trt_bf16_enable' in TensorRT EP on GPU
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I'm using ONNX Runtime with TensorRT Execution Provider on GPU. I noticed that 'trt_fp16_enable' is supported and works well, but there is no documentation regarding 'trt_bf16_enable'.
- Is there any plan to support BF16 via 'trt_bf16_enable' in the near future?
- Are there any known workarounds or experimental builds for BF16 support on GPU using TensorRT EP?
- If it's not yet supported, is it on the roadmap?
### Describe scenario use case
We are running SLM model on A100 with TensorRT EP, and would like to use BF16 for better performance compared to FP32, while keeping accuracy close.
Contributor guide
Research direction
The request concerns the TensorRT Execution Provider and the trt_bf16_enable option, but names no implementation files or tests. Start by checking the provider's existing trt_fp16_enable support and documentation, then determine the scope of BF16 support. Done would require a decided support path with corresponding implementation and validation details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100