How to quantize Linear/LN/ReLU-like structures with int8.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
My TensorRT version is 8.6.10 on Orin.
My model is Linear/LN/ReLU-like structure as below:
I add Q/DQ nodes before MatMul node to do INT8 as below:
However, INT8 is slower than FP16.
I draw the INT8 engine figure as below.
What is the best practice for quantize Linear/LN/ReLU-like structures? which takes about 50% latency in my model.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported TensorRT 8.6.10 setup on Orin, the Linear/LN/ReLU-like graph, and the Q/DQ placement before MatMul shown in the issue. Compare the INT8 and FP16 engine figures and quantify the latency of the affected structure; done means identifying a validated quantization approach that avoids the reported INT8 slowdown.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100