NVIDIA / NVIDIA/TensorRT

After upgrading from 8.6 to 10.8 or 10.9, tensorrt's results are inconsistent with onnxrt

Open
#4,400 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

internal-bug-tracked Module:Accuracy triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

After upgrading tensorrt to 10.8, the model accuracy decreased.
After setting all nodes of the model to output, the model accuracy was aligned. It was suspected that the fusion strategy introduced by the upgrade caused the accuracy problem.
Finally, we used the polygraphy tool to find an onnx subgraph that could reproduce the problem.
By the way, it needs to be reminded that the model only uses fp32 precision, not even fp16 or int8

Environment

TensorRT Version: 10.8 & 10.9

NVIDIA GPU: 3090

NVIDIA Driver Version: 550.67

CUDA Version: 12.2

Relevant Files

I post onnx file here:
Model link: https://github.com/2730gf/issues/blob/main/trt_inconsistent/mini_graph.onnx

Steps To Reproduce

Commands or scripts:
polygraphy run mini_graph.onnx -v -v -v -v -v --pool-limit workspace:20G --onnxrt --trt --validate --atol 1e-4 --rtol 1e-3 --onnx-outputs p2o.Concat.125 --trt-outputs p2o.Concat.125
Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked mini_graph.onnx and run the supplied Polygraphy command using TensorRT and ONNX Runtime. Compare the outputs at p2o.Concat.125 under TensorRT 10.8/10.9; done means the precision discrepancy is reproduced and its TensorRT fusion cause is identified or corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.