microsoft / microsoft/onnxruntime
TensorRT EP API for environment variables (esp, on FP16, INT8)
@stevenlix is already working on this.
Since Mar 3, 2021.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
For various environment variables on TensorRT EP, ONNXRT needs to provide an API to override settings per model based.
The most critical environment variables are FP16 (ORT_TENSORRT_FP16_ENABLE) and INT8 (ORT_TENSORRT_INT8_ENABLE) for performance.
This will provide scalability and generality, for example, on Triton Inference server. TensorFlow has an API for FP16 support.
This issue was originated from Huggingface when they run Triton experiments on ONNXRT, TensorFlow, TensorRT backends.
(See https://github.com/triton-inference-server/onnxruntime_backend/issues/20#issuecomment-778686795: Triton-ONNXRT model’s config needs to be able to enable FP16 inferencing (like Triton-TensorFlow) as a 1st priority so that there is no server downtime without restarting with docker run -e ORT_TENSORRT_FP16_ENABLE=1)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.