NVIDIA / NVIDIA/TensorRT

[QUESTION] Limitation of TensorRT I/O format when using a Cutlass convolution

Open
#3,192 5 comments 0 reactions 1 assignee View on GitHub

@zerollzeng is already working on this.

Since Aug 9, 2023.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Hello,

This is not so much an issue with a particuliar TensorRT version as much a general compatibility issue between two NVIDIA software.
According to the documentation of TensorRT the only supported I/O format for INT8 data types are either kLinear or kCHWx. As I/O also applies to passed passed to and from plugins, this means an INT8 plugin should have this data format to avoid reformatting.
Now, I mainly want to leverage Cutlass in my plugin, and they have confirmed to me that their ImplicitGemm convolutions only support channels as the leading dimension. Plus, its still unclear that I can use bias when using an interleaved format, so my only real option right now is to use NHWC.
But this is unfortunate, as both these products are NVIDIA-made.

Is there any way to pass INT8+kHWC data to a plugin in TensorRT? I'm open to any tricks you might have. Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.