nan output in fp16 mode of TensorRT 10.16 when running trtexec on GPU RTX 5080
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
trtexec --onnx=camera.backbone.onnx --saveEngine=camera.backbone.engine
trtexec --onnx=camera.backbone.onnx --fp16 --saveEngine=camera.backbone_fp16.engine
trtexec --loadEngine=camera.backbone_fp16.engine --loadInputs=img:img.bin,depth:depth.bin --exportOutput=output_fp16.json
trtexec --loadEngine=camera.backbone.engine --loadInputs=img:img.bin,depth:depth.bin --exportOutput=output.json
output.json is correct while output_fp16.json are all nan values
Environment
RTX 5080
TRT 10.16
Driver Version: 580.142
Cuda version 13.0
Operating System: Ubuntu 22.04
Model link : https://drive.google.com/file/d/1JpS_gWXe_a96EIFh5DZmYihUS-lUZnjt/view?usp=drive_link
input bin files can be generated randomly
import numpy as np
np.random.seed(42)
img_data = np.random.rand(1, 2, 3, 256, 704).astype(np.float32)
depth_data = np.random.rand(1, 2, 1, 256, 704).astype(np.float32)
img_data.tofile('img.bin')
depth_data.tofile('depth.bin')
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the listed trtexec commands with TensorRT 10.16 on an RTX 5080, using the linked model and the generated img.bin and depth.bin inputs. Compare output.json with output_fp16.json and inspect the trtexec FP16 engine path. Done means the cause is isolated and FP16 inference no longer produces all-NaN output, or the issue is narrowed to a documented environment or model condition.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100