Wrong encoder output for NeMo Parakeet (FastConformer) on RTX 5090 (sm_120a) — all precisions, all opt levels, TRT 10.14 and 11.1

Description

The encoder of NVIDIA's own parakeet-tdt-0.6b-v3 (NeMo FastConformer, standard ONNX export)
produces numerically wrong output on an RTX 5090 (sm_120a) with every TensorRT configuration
we tried, while onnxruntime-CUDA on the same GPU matches the fp32 CPU reference to ~7e-7. The
corruption is deterministic (bit-identical across runs) and severe enough that downstream TDT
decoding emits an empty transcript. The same ONNX + same TensorRT version produced correct
engines on an RTX 3090 (sm_86) — only the target architecture changed.

Cosine similarity of the encoder output vs the fp32 onnxruntime reference (1.0 = correct;
values below computed on identical input, ref_mel_jw16k.npy, shapes (1,128,1201)):

Additional signatures:

• 114 of 151 output frames diverge; 37 are correct — a partial-frame pattern suggestive of an
  attention-mask/window handling fault rather than accumulated numeric error.
• The 10.14 engines also report encoded_lengths = 149 instead of the correct 151
  (11.1 gets 151, output still wrong).
• Fresh timing caches / --dumpOptimizationProfile variations do not change the result;
  outputs are bit-identical run to run.
• Verified through three independent I/O paths (two independently written Python harnesses and
  trtexec --loadInputs/--exportOutput): identical results per engine, so measurement error
  is excluded.

Environment

• GPU: NVIDIA GeForce RTX 5090 (sm_120a)
• Driver: 595.71.05 (open kernel modules)
• CUDA: 13.0 (nvcc release 13.0)
• TensorRT: 10.14.1.48 (apt, +cuda13.0) and 11.1.0.106 (pip wheel) — both affected
• OS: Ubuntu 24.04 x86_64
• Model: parakeet-tdt-0.6b-v3 encoder, ONNX export via NeMo (fp32, external weights;
  input audio_signal (1,128,T) fp32 + length int64; outputs outputs (1,1024,T/8) fp32 +
  encoded_lengths int64)

Reproduction

Attached: ref_mel_jw16k.npy (reference input), ref_encout_ort.npy (fp32 ORT reference
output), build_and_compare_trt.py (builds the engine from the ONNX with the TRT Python API and
prints the cosine; TRT_OPT_LEVEL env selects builder optimization level), compare_ort.py
(the onnxruntime side). The encoder ONNX itself is ~2.4 GB and is reproducible with NeMo's
standard export of nvidia/parakeet-tdt-0.6b-v3; we can supply our exact file on request.

trtexec-only repro (no Python in the execution path):

trtexec --onnx=encoder-parakeet-tdt-0.6b-v3.onnx \
        --saveEngine=enc.plan --shapes=audio_signal:1x128x1201,length:1
trtexec --loadEngine=enc.plan --shapes=audio_signal:1x128x1201,length:1 \
        --loadInputs=audio_signal:audio_signal.bin,length:length_i64.bin \
        --exportOutput=out.json --iterations=1 --warmUp=0 --duration=0
# cosine(out.json vs ref_encout_ort.npy) => 0.35-0.83 depending on precision; expected ~1.0

Possibly related

• #4813 — TRT 11.1 strongly-typed fp32 silently wrong vs ONNXRuntime on sm_120 (RTX 5070 Ti),
  deformable-attention (DETR-family) model; reported correct on TRT 10.13. Same
  silent-accuracy-on-Blackwell class; our case additionally reproduces on 10.14.
• #4743 / #4715 — sm_120 Myelin faults (crashes rather than silent corruption).

Expected behavior

Encoder output numerically equivalent to the onnxruntime fp32 reference (as it is when the same
ONNX is built for sm_86, and as onnxruntime-CUDA achieves on the same sm_120a GPU).

Start with the attached build_and_compare_trt.py and compare_ort.py, using ref_mel_jw16k.npy and ref_encout_ort.npy as references. Reproduce the result with the provided trtexec commands and compare behavior across the listed TensorRT versions, precisions, and optimization levels. Done means the sm_120a encoder output matches the ONNX Runtime reference and encoded_lengths is correct.