Pointers for TensorRT model with uint8/int8 input
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
By using pytorch-quantization i was able to create TensorRT engine models that are (almost) fully int8 and have lower latencies than FP16 equivalent models.
One of the downsides is that the input is reformatted from FP32 to INT8 for the next 2D Conv. This takes up to 5% of the total latency budget of the model. On the other hand, I need to cast my images from uint8/int8 to FP32. As you can see this is not efficient and introduces double casting + transferring 4x more memory to the GPU for inference, which is not needed since the model is fully int8.
Is there a possibility to create engines that have (u)int8 input? And do I need to adopt anything during quantization?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked pytorch-quantization documentation and TensorRT's engine input handling to determine whether uint8 or int8 inputs are supported. Confirm whether quantization requires changes, and document the supported path or the missing capability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100