Set layer precision still doesn't take effect in TensorRT 8.6.1.
@zerollzeng is already working on this.
Since Aug 17, 2023.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
As I had reflected in this Skipping tactic 0x0000000000000000 due to Myelin error" degrade performance.,set layer precision may failed in TensorRT 8.4.3 due to the ConstShuffleFusion.
In these days, I try TensorRT 8.6.1, but it seems that setting layer precision may still fail due to the ConstShuffleFusion.
For example, as show in the graph, the Max op take a const input named "phase0_tf/predict_node/y:0", and the value seem to be fp16 subnormal, so I use set_precision api to set the layer ("phase0_tf/predict_node/y:0") to fp32 explicitly.
The verbose logs are as follows:
When the fp16 subnormal is not set to fp32, the log are as follows, the layer "phase0_tf/predict_node/y:0 + (Unnamed Layer* 522) [Shuffle]" is fp16 precision:
However, when the fp16 subnormal is set to fp32, the layer "phase0_tf/predict_node/y:0 + (Unnamed Layer* 522) [Shuffle]" is still fp16.
By the way, the ConstShuffleFusion produce two kind of layer, such as
I am confused the differences. Is that the reason set_precision fails of the layer "phase0_tf/predict_node/y:0"?
Looking forward to your reply. Thanks a lot!
Environment
TensorRT Version: 8.6.1
NVIDIA GPU: T4
NVIDIA Driver Version: 510
CUDA Version:12.0
CUDNN Version:
Operating System: Ubuntu20.04
Python Version (if applicable):
Tensorflow Version (if applicable): 1.4
PyTorch Version (if applicable):
Baremetal or Container (if so, version):
Relevant Files
Model link:
Steps To Reproduce
Commands or scripts:
Have you tried the latest release?:
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.