NVIDIA / NVIDIA/TensorRT

Read the output shape error in profile from the engine file

Open
#3,891 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

I used the trtexec and polygraphy commands to export from onnx to trt, and the model output was fine, but the engine file output shape I read in the code did not match that in the log.

Environment

Use a mirror nvidia/cuda:12.1.0-devel-ubuntu20.04.

TensorRT Version: 10.0.1.6

NVIDIA GPU: L4

NVIDIA Driver Version: 535.129.03

CUDA Version: 12.1.0

CUDNN Version:

Operating System: ubuntu20.04

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:
https://drive.google.com/file/d/197HgyO3Dwua89QERaDfYIx-hNhNpUKJJ/view?usp=drive_link

Steps To Reproduce

Commands or scripts:
polygraphy run ./weights/parseq_ar_decoder_bs1_smi.onnx --onnxrt --trt --pool-limit workspace:1G --save-engine=./weights/parseq_ar_decoder_bs1_smi.trt --trt-min-shapes tgt:[1,1] --trt-opt-shapes tgt:[1,41] --trt-max-shapes tgt:[1,251] --atol 1e-3 --rtol 1e-3 --verbose
/TensorRT-10.0.1.6/bin/trtexec --onnx=./weights/parseq_ar_step_decoder_bs1_smi.onnx --saveEngine=./weights/parseq_ar_step_decoder_bs1_smi.trt --minShapes=tgt:1x1 --optShapes=tgt:1x1 --maxShapes=tgt:1x251

Have you tried the latest release?:
yes.

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):
I used python and onnx inference and the results were normal.

Use the code in the link below to allocate memory.
https://github.com/NVIDIA/TensorRT/blob/6d2fa4df7fa3e3f4bf5dd27586ba053b4ae57cd5/samples/python/common_runtime.py#L92
polygraphy
image

trtexec
image

In the code:
image

The source code is the decoder part of parseq, and I refactored it to use only the ar part of it (decode_ar=True, refine_iters=False).
https://github.com/baudm/parseq/blob/1902db043c029a7e03a3818c616c06600af574be/strhub/models/parseq/model.py#L86

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the referenced common_runtime.py allocation code around line 92, then reproduce the commands using the provided TensorRT, CUDA, GPU, and model details. Compare the output shapes reported by polygraphy, trtexec, and the code; the issue is complete when the discrepancy is explained and the engine shape handling is corrected or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.