TRT 11.1 strongly-typed FP32 engine silently produces wrong results for a DETR-style detection model (correct in ONNXRuntime and TRT 10.13)

Description

Building a strongly-typed engine from an FP32 ONNX of a D-FINE (DETR-family, deformable-attention decoder) detection model on TensorRT 11.1.0.106 produces silently wrong outputs: detection scores diverge from ONNXRuntime by up to 0.15 (absolute, on sigmoid outputs) and output boxes by up to 597 px, where FP32-vs-FP32 agreement should be ~1e-4. The build completes with no warnings or errors.

The same ONNX file is exact in ONNXRuntime, and the same model executes correctly on TensorRT 10.13.3.9. On a real dataset (VisDrone test set) the mis-execution collapses recall 0.49 -> 0.34 and F1 0.587 -> 0.458.

Repo where issue was encountered - D-FINE-seg

Environment

TensorRT Version: 11.1.0.106 (tensorrt / tensorrt-cu13 wheels from PyPI)

NVIDIA GPU: GeForce RTX 5070 Ti, 16 GB (sm_120)

NVIDIA Driver Version: 580.159.03

CUDA Version: 13.0

CUDNN Version: 9.20.0.48 (not used by the repro; TRT wheels vendor their own deps)

Operating System: Ubuntu 22.04.5 LTS, kernel 6.8.0-124-generic

Python Version (if applicable): 3.13.12

Tensorflow Version (if applicable): —

PyTorch Version (if applicable): 2.12.1+cu130 (used only for CUDA buffers in the repro script; model is loaded from ONNX)

Baremetal or Container (if so, version): baremetal, pip wheels (pip install tensorrt==11.1.0.106 onnxruntime==1.21.1 onnx==1.21.0)

Relevant Files

Model link: https://drive.google.com/file/d/1Fs6nh2Edv1-1xFbqnIN5zTvE4nzfe7Ta/view?usp=share_link

• model.onnx — D-FINE-S detection model (HGNetv2 backbone + hybrid encoder + 3-layer deformable-attention decoder + fused postprocessor), FP32, static [1,3,640,640] input, outputs labels [1,300] int64, boxes [1,300,4], scores [1,300]. Exported with torch.onnx.export (dynamo, opset 19), simplified with onnxsim.
• The bug also reproduces with a TorchScript-exporter (dynamo=False, opset 17) export of the same weights — it is not specific to one exporter's graph flavor.
• repro.py — self-contained comparison script (below), deterministic synthetic input, no dataset needed.

Steps To Reproduce

Commands or scripts:

pip install tensorrt==11.1.0.106 onnxruntime==1.21.1 onnx==1.21.0 numpy torch
python repro.py model.onnx

repro.py:

import sys
import numpy as np
import onnxruntime as ort
import tensorrt as trt
import torch

onnx_path = sys.argv[1]

# deterministic structured input (gradient + rectangles, image-like, in [0,1])
rng = np.random.default_rng(0)
x = np.tile(np.linspace(0.2, 0.8, 640, dtype=np.float32)[None, :], (640, 1))
img = np.stack([x, x.T, 0.5 * (x + x.T)], 0)
for _ in range(40):
    x0, y0 = rng.integers(0, 560, 2)
    w, h = rng.integers(20, 80, 2)
    img[:, y0 : y0 + h, x0 : x0 + w] = rng.random(3, dtype=np.float32)[:, None, None]
inp = img[None]  # [1,3,640,640]

# ONNXRuntime reference
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
ort_out = dict(zip([o.name for o in sess.get_outputs()], sess.run(None, {"input": inp})))

# TensorRT: strongly-typed engine from the same FP32 ONNX
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network()  # strongly typed (TRT >= 11 default)
parser = trt.OnnxParser(network, logger)
assert parser.parse(open(onnx_path, "rb").read())
config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 2 << 30)
engine = trt.Runtime(logger).deserialize_cuda_engine(
    builder.build_serialized_network(network, config)
)
ctx = engine.create_execution_context()

bufs = {}
for i in range(engine.num_io_tensors):
    name = engine.get_tensor_name(i)
    shape = tuple(engine.get_tensor_shape(name))
    dt = {"FLOAT": torch.float32, "INT64": torch.int64, "HALF": torch.float16}[
        engine.get_tensor_dtype(name).name
    ]
    bufs[name] = torch.empty(shape, dtype=dt, device="cuda")
    ctx.set_tensor_address(name, bufs[name].data_ptr())
bufs["input"].copy_(torch.from_numpy(inp))
ctx.execute_async_v3(torch.cuda.current_stream().cuda_stream)
torch.cuda.synchronize()

for name in ("scores", "boxes"):
    ref, out = ort_out[name].ravel(), bufs[name].float().cpu().numpy().ravel()
    print(f"{name:7s} max_abs_diff={np.abs(ref - out).max():.4f}")
print("scores top5 ORT:", np.round(np.sort(ort_out["scores"].ravel())[::-1][:5], 4))
print("scores top5 TRT:", np.round(np.sort(bufs["scores"].float().cpu().numpy().ravel())[::-1][:5], 4))

Observed output (RTX 5070 Ti, driver 580.159.03, TRT 11.1.0.106):

scores  max_abs_diff=0.1495
boxes   max_abs_diff=597.0000
scores top5 ORT: [0.1603 0.1174 0.0951 0.0739 0.0698]
scores top5 TRT: [0.2487 0.2349 0.2262 0.2203 0.2194]

Expected: max_abs_diff ~1e-4 or less (FP32 vs FP32). The engine builds without any warning; the wrong results are silent.

Have you tried the latest release?: Yes — 11.1.0.106 is the newest TensorRT on PyPI at the time of filing. The model executes correctly on TensorRT 10.13.3.9 (weak-typed build; FP16 engine matches the PyTorch F1 on our full benchmark).

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes — ONNXRuntime (CPU EP) executes the file exactly (it is the reference in the repro). PyTorch source model agrees with ONNXRuntime.

Additional context

• The mis-execution appears tied to FP32 compilation of the transformer decoder region. Related observations on the same model, all silent:
  • A mixed engine authored as FP16 backbone/encoder + FP32 decoder (explicit Cast boundaries, strongly typed) collapses the same way (dataset F1 0.587 -> 0.445), while an all-FP16 engine (only Softmax kept FP32) is essentially correct (F1 0.584–0.586).
  • Inserting FP32 islands (~100 nodes of box-decode arithmetic) into the otherwise-FP16 graph measurably degrades accuracy instead of improving it.
  • With FP16 engines, accuracy differs between two ONNX graph flavors of the same weights (torch dynamo exporter 0.584 vs TorchScript exporter 0.586 dataset F1) — consistent with a graph-compilation sensitivity rather than kernel precision.
• All converted/authored ONNX variants used in these experiments execute identically to the FP32 reference in ONNXRuntime, so the divergence is introduced at TensorRT build/execution time.

Start by running repro.py with model.onnx under TensorRT 11.1.0.106 and ONNXRuntime to confirm the score and box divergence. Compare the strongly typed build path with the known-correct TensorRT 10.13.3.9 behavior and inspect the transformer decoder region. Done means the same FP32 model produces results within roughly 1e-4 without silent errors.