NVIDIA / NVIDIA/TensorRT

nan output in fp16 mode of TensorRT 10.16 when running trtexec on GPU RTX 5080

Open
#4,748 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:Samples
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

trtexec --onnx=camera.backbone.onnx --saveEngine=camera.backbone.engine
trtexec --onnx=camera.backbone.onnx --fp16 --saveEngine=camera.backbone_fp16.engine

trtexec --loadEngine=camera.backbone_fp16.engine --loadInputs=img:img.bin,depth:depth.bin --exportOutput=output_fp16.json

trtexec --loadEngine=camera.backbone.engine --loadInputs=img:img.bin,depth:depth.bin --exportOutput=output.json

output.json is correct while output_fp16.json are all nan values

Environment

RTX 5080

TRT 10.16

Driver Version: 580.142

Cuda version 13.0

Operating System: Ubuntu 22.04

Model link : https://drive.google.com/file/d/1JpS_gWXe_a96EIFh5DZmYihUS-lUZnjt/view?usp=drive_link

input bin files can be generated randomly

import numpy as np

np.random.seed(42)
img_data = np.random.rand(1, 2, 3, 256, 704).astype(np.float32)

depth_data = np.random.rand(1, 2, 1, 256, 704).astype(np.float32)

img_data.tofile('img.bin')
depth_data.tofile('depth.bin')

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the listed trtexec commands with TensorRT 10.16 on an RTX 5080, using the linked model and the generated img.bin and depth.bin inputs. Compare output.json with output_fp16.json and inspect the trtexec FP16 engine path. Done means the cause is isolated and FP16 inference no longer produces all-NaN output, or the issue is narrowed to a documented environment or model condition.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.