NVIDIA / NVIDIA/TensorRT

wrong results of TensorRT 10.0 when running on GPU Tesla T4

Open
#3,999 38 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

internal-bug-tracked Module:Accuracy triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

The output of the TensorRT 10 model converted from ONNX is incorrect, while the output of the TensorRT 8.6 model is correct. The issue seems to be located in some fully connected layers in the TensorRT 10 model, where the error in the output suddenly becomes very large. The exact cause is unknown. Please help to resolve this issue.

Environment

TensorRT Version: TensorRT 10.0.1

NVIDIA GPU: Tesla T4

NVIDIA Driver Version: 450.36.06

CUDA Version: 11.0

CUDNN Version:8.0.0

Operating System:

onnx opset17

Relevant Files

Model link: https://drive.google.com/file/d/1QBbmtdaecWAHzqMdh10QVbdSjTWzleqo/view?usp=sharing

Steps To Reproduce

  1. Convert the ONNX model to TensorRT 10 using ./trtexec --onnx=./test.onnx --device=0 --saveEngine=./test.trtmodel --precisionConstraints=obey.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked model and the ./trtexec command using test.onnx, then compare its TensorRT 10.0.1 output with the TensorRT 8.6 result on the Tesla T4. Inspect the fully connected layers where the error becomes large; done means identifying the cause and resolving the incorrect inference output.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.