[QUESTION] Limitation of TensorRT I/O format when using a Cutlass convolution
@zerollzeng is already working on this.
Since Aug 9, 2023.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Hello,
This is not so much an issue with a particuliar TensorRT version as much a general compatibility issue between two NVIDIA software.
According to the documentation of TensorRT the only supported I/O format for INT8 data types are either kLinear or kCHWx. As I/O also applies to passed passed to and from plugins, this means an INT8 plugin should have this data format to avoid reformatting.
Now, I mainly want to leverage Cutlass in my plugin, and they have confirmed to me that their ImplicitGemm convolutions only support channels as the leading dimension. Plus, its still unclear that I can use bias when using an interleaved format, so my only real option right now is to use NHWC.
But this is unfortunate, as both these products are NVIDIA-made.
Is there any way to pass INT8+kHWC data to a plugin in TensorRT? I'm open to any tricks you might have. Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.