How to specify inputs fmt when using trtexec in my case?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Need help about the usage of trtexec.
my trtexec version is on TensorRT V9.3 &&&& RUNNING TensorRT.trtexec [TensorRT v9300]
I have a pytorch model trained with AMP bfloat16 type . the model exported into onnx fp32 format, and then I use trtexec above to convert the onnx fp32 into trt fp16.
the model have six inputs, which are:
x dtype float32 shape_as: batchsize X seq_len X channels;
mask dtype bool shape_as: batchsize X seq_len
text dtype int32 shape_as: batchsize X seq_len_text;
t dtype float32 shape_as: batchsize,
cond1 dtype float32 shape_as: batchsize X cond1_channels;
cond2 dtype float32 shape_as: batchsize X seq_len X channels
and one output as follow:
estimator_output dtype float32 shape_as: batch_size X seq_len X channels;
the seq_len, and seq_len_text are dynamic dims specified in onnx exportation.
dynamic_axes = {
"x": {1, 'seq_len'},
"mask": {1: 'seq_len'},
"text": {1: "seq_len_text"},
"cond2": {1: "seq_len"},
"estimator_output": {1: "seq_len"}
}
Now I am writing the trtexec command to convert onnx fp32 into tensorrt fp16. I have to use --inputIOFormats to specify each input. the fmt part can be specified as chw, chw2, chw16 ..., but my input set channels as the last dim, and there is no such btc fomart in the fmt options, which one should I set in my case?
Another problem is, I had tried to convert the onnx fp32 above to bf16 tensorrt engine. the command is as follows:
/bin/trtexec --onnx=MyFp32.onnx --saveEngine=./resultengine_bf16.engine --bf16 --minShape=x:2x4x80,mask:2x4,text:2x2,cond2:2x4x80 --optShape=x:2x100x80,mask:2x100,text:2x50,cond2:2x100x80 --maxShapes=x:2x1000x80,mask:2x1000,text:2x500,cond2:2x1000x80 --inputIOFormats=fp16:chw,bool:chw,int32:chw,fp16:chw,fp16:chw,fp16:chw --outputIOFormats=fp16:chw
During execution, it report error like:
Error: Error: 9 skipping tactic 0x0000000000 due to exception Assertion type == myelinTypeInt32 .....
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied trtexec command, especially --inputIOFormats and the dynamic-shape options, and consult the TensorRT 9.3 trtexec usage documentation. Reproduce the BF16 build and investigate the reported myelinTypeInt32 assertion; done means the input formats and shapes are validated or the failure is confirmed as a TensorRT issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch
- Domain
- machine-learning, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100