NVIDIA / NVIDIA/TensorRT

W4A16 FP4 build succeeds while silently realizing FP32 (no warning, no refusal)

Open
#4,833 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:Quantization
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Summary

A W4A16 build succeeds while silently dequantizing FP4 weights to FP32. Nothing in the build output indicates the requested precision was not realized, so a user can benchmark an "FP4 engine" that is executing FP32.

Affects: TensorRT 10.16, still observed on 11.2.1.

Detail

Feeding a genuine FLOAT4E2M1 weight-QDQ graph to TensorRT:

  • W4A4 fails loudly ("type not supported yet") — this is fine and is the correct behaviour.
  • W4A16 succeeds and silently realizes FP32.

We only detected this because our device payload records precision_realized independently of the requested precision. Two independent observations on the same engine: the manifest reports realized precisions FP32, INT8 for a graph requested as fp4, and the same engine subsequently fails to complete a trtexec --loadEngine timing run.

The ask

Not an FP4 Conv kernel. The request is that the builder warn or refuse when an explicitly quantized precision is discarded, exactly as it already refuses W4A4. A silent downcast to FP32 is a correctness trap: it converts a precision request into a performance measurement of something else entirely.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the TensorRT builder path handling FLOAT4E2M1 weight-QDQ graphs and compare W4A16 with the existing W4A4 rejection. Reproduce with the described graph, inspect the build output and realized precision, then use trtexec --loadEngine timing to confirm that completion means warning or refusal rather than a silent FP32 engine.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.