NVIDIA / NVIDIA/TensorRT

How to specify inputs fmt when using trtexec in my case?

Open
#4,679 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:Samples
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Need help about the usage of trtexec.
my trtexec version is on TensorRT V9.3 &&&& RUNNING TensorRT.trtexec [TensorRT v9300]
I have a pytorch model trained with AMP bfloat16 type . the model exported into onnx fp32 format, and then I use trtexec above to convert the onnx fp32 into trt fp16.
the model have six inputs, which are:

x                        dtype float32  shape_as:  batchsize X seq_len X channels;   
mask                 dtype bool      shape_as:  batchsize X seq_len
text                   dtype int32      shape_as:  batchsize X  seq_len_text;
t                        dtype float32  shape_as:   batchsize,
cond1               dtype float32   shape_as:   batchsize X cond1_channels;
cond2               dtype float32  shape_as:    batchsize X seq_len X channels

and one output as follow:

estimator_output    dtype float32    shape_as:  batch_size X seq_len X channels;

the seq_len, and seq_len_text are dynamic dims specified in onnx exportation.

    dynamic_axes = {
       "x": {1, 'seq_len'},
      "mask": {1: 'seq_len'},
      "text": {1: "seq_len_text"},
     "cond2": {1: "seq_len"},
     "estimator_output": {1: "seq_len"}
    }

Now I am writing the trtexec command to convert onnx fp32 into tensorrt fp16. I have to use --inputIOFormats to specify each input. the fmt part can be specified as chw, chw2, chw16 ..., but my input set channels as the last dim, and there is no such btc fomart in the fmt options, which one should I set in my case?

Another problem is, I had tried to convert the onnx fp32 above to bf16 tensorrt engine. the command is as follows:

/bin/trtexec --onnx=MyFp32.onnx --saveEngine=./resultengine_bf16.engine --bf16 --minShape=x:2x4x80,mask:2x4,text:2x2,cond2:2x4x80 --optShape=x:2x100x80,mask:2x100,text:2x50,cond2:2x100x80 --maxShapes=x:2x1000x80,mask:2x1000,text:2x500,cond2:2x1000x80 --inputIOFormats=fp16:chw,bool:chw,int32:chw,fp16:chw,fp16:chw,fp16:chw --outputIOFormats=fp16:chw

During execution, it report error like:

Error: Error: 9  skipping tactic 0x0000000000 due to exception Assertion type == myelinTypeInt32 .....

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied trtexec command, especially --inputIOFormats and the dynamic-shape options, and consult the TensorRT 9.3 trtexec usage documentation. Reproduce the BF16 build and investigate the reported myelinTypeInt32 assertion; done means the input formats and shapes are validated or the failure is confirmed as a TensorRT issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.