NVIDIA / NVIDIA/TensorRT

How to debug trt fp16 Nan output with polygraphy?

Open
#4,502 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:Performance Module:Polygraphy Module:Quantization
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

I used polygraphy run model.onnx --trt --fp16 --precision-constraints none --data-loader-script loader.py -v --validate --fail-fast to check fp16 onnx' outputs (model.onnx is fp32) and got the following error.

[I] Output Validation | Runners: ['trt-runner-N0-06/30/25-19:45:22']
[I]     trt-runner-N0-06/30/25-19:45:22     | Validating output: pred_multipath (check_inf=True, check_nan=True)
[I]         mean=nan, std-dev=nan, var=nan, median=nan, min=nan at (0, 0, 0, 0), max=nan at (0, 0, 0, 0), avg-magnitude=nan, p90=nan, p95=nan, p99=nan
[V]             Could not generate histogram. Note: Error was: autodetected range of [nan, nan] is not finite
[V]             
[E]         NaN Detected | One or more NaNs were encountered in this output
[I]         Note: Use -vv or set logging verbosity to EXTRA_VERBOSE to display locations of NaNs
[E]         Inf Detected | One or more non-finite values were encountered in this output
[I]         Note: Use -vv or set logging verbosity to EXTRA_VERBOSE to display non-finite values
[E]         FAILED | Errors detected in output: pred_multipath
[I]     trt-runner-N0-06/30/25-19:45:22     | Validating output: path_prob (check_inf=True, check_nan=True)
[I]         mean=nan, std-dev=nan, var=nan, median=nan, min=nan at (0, 0), max=nan at (0, 0), avg-magnitude=nan, p90=nan, p95=nan, p99=nan
[V]             ---- Values ----
                    [[nan nan nan nan nan]]
[V]             Could not generate histogram. Note: Error was: autodetected range of [nan, nan] is not finite
[V]             
[E]         NaN Detected | One or more NaNs were encountered in this output
[E]         Inf Detected | One or more non-finite values were encountered in this output
[E]         FAILED | Errors detected in output: path_prob
[I]     trt-runner-N0-06/30/25-19:45:22     | Validating output: pred_target_agent_attribute (check_inf=True, check_nan=True)
[I]         mean=20.279, std-dev=30.123, var=907.39, median=3.0488, min=1.7881e-07 at (0, 3), max=80.562 at (0, 0), avg-magnitude=20.279, p90=63.444, p95=72.003, p99=78.851
[V]             ---- Values ----
                    [[8.0562500e+01 5.2031250e+01 7.6293945e-05 1.7881393e-07 3.0488281e+00
                      4.5117188e+00 1.8007812e+00]]
[V]             ---- Histogram ----
                Bin Range        |  Num Elems | Visualization
                (1.79e-07, 8.06) |          5 | ########################################
                (8.06    , 16.1) |          0 | 
                (16.1    , 24.2) |          0 | 
                (24.2    , 32.2) |          0 | 
                (32.2    , 40.3) |          0 | 
                (40.3    , 48.3) |          0 | 
                (48.3    , 56.4) |          1 | ########
                (56.4    , 64.4) |          0 | 
                (64.4    , 72.5) |          0 | 
                (72.5    , 80.6) |          1 | ########
[I]         PASSED | Output: pred_target_agent_attribute is valid
[I]     trt-runner-N0-06/30/25-19:45:22     | Validating output: pred_scores (check_inf=True, check_nan=True)
[I]         mean=0.092712, std-dev=0, var=0, median=0.092712, min=0.092712 at (0,), max=0.092712 at (0,), avg-magnitude=0.092712, p90=0.092712, p95=0.092712, p99=0.092712
[V]             ---- Values ----
                    [0.0927124]
[V]             ---- Histogram ----
                Bin Range            |  Num Elems | Visualization
                (-0.407  , -0.307  ) |          0 | 
                (-0.307  , -0.207  ) |          0 | 
                (-0.207  , -0.107  ) |          0 | 
                (-0.107  , -0.00729) |          0 | 
                (-0.00729, 0.0927  ) |          0 | 
                (0.0927  , 0.193   ) |          1 | ########################################
                (0.193   , 0.293   ) |          0 | 
                (0.293   , 0.393   ) |          0 | 
                (0.393   , 0.493   ) |          0 | 
                (0.493   , 0.593   ) |          0 | 
[I]         PASSED | Output: pred_scores is valid
[I]     trt-runner-N0-06/30/25-19:45:22     | Validating output: pred_ttc (check_inf=True, check_nan=True)
[I]         mean=4, std-dev=0, var=0, median=4, min=4 at (0,), max=4 at (0,), avg-magnitude=4, p90=4, p95=4, p99=4
[V]             ---- Values ----
                    [4.]
[V]             ---- Histogram ----
                Bin Range  |  Num Elems | Visualization
                (3.5, 3.6) |          0 | 
                (3.6, 3.7) |          0 | 
                (3.7, 3.8) |          0 | 
                (3.8, 3.9) |          0 | 
                (3.9, 4  ) |          0 | 
                (4  , 4.1) |          1 | ########################################
                (4.1, 4.2) |          0 | 
                (4.2, 4.3) |          0 | 
                (4.3, 4.4) |          0 | 
                (4.4, 4.5) |          0 | 
[I]         PASSED | Output: pred_ttc is valid
[E]     FAILED | Output Validation

I used polygraphy run model.onnx --trt --fp16 --precision-constraints none --data-loader-script loader.py -v --validate --fail-fast --trt-outputs mark all --save-outputs outputs.json to get layerwise outputs but got the following error:

[E] 2: [myelinBuilderUtils.cpp::operator()::751] Error Code 2: Internal Error ([ShapeHostToDeviceCopy 0] requires bool or uint8 I/O but node can not be handled by Myelin. Operation is not supported.)
[!] Invalid Engine. Please ensure the engine was built correctly

It says --trt outputs mark all may disable TensorRT's layer fusion, resulting in performance degradation or errors (such as Myelin optimizer errors). Can I get the layerwise outputs from trt model? How can I debug this Nan issue?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Polygraphy command using model.onnx and loader.py, including the --trt-outputs mark all variant that writes outputs.json. Start with the reported NaN validation and ShapeHostToDeviceCopy error, then determine whether a reproducible TensorRT or Polygraphy issue remains. Done means a documented cause and debugging path, or a confirmed fix with validation results.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.