Decode latency of FP4 LLM is higher than expected

System Info

GPU: RTX 5090

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

TensorRT-LLM version: 0.20.0rc2
TensorRT-Model-Optimizer version: 0.29.0

Code: TensorRT-Model-Optimizer LLM PTQ example
https://github.com/NVIDIA/TensorRT-Model-Optimizer/tree/main/examples/llm_ptq
--quant type: nvfp4

Expected behavior

FP4 should be faster than FP8

actual behavior

At prefill stage, FP4 shows faster(almost doubled) inference compared to FP8
But at decode stage, FP4 is slightly slower than FP8

additional notes

I am attempting to run LLM inference using NVFP4 on an RTX 5090.

I'm using example llm_ptq (quantize, build, benchmark) code from TensorRT-Model-Optimizer.
(Only changed --quant datatype into "nvfp4")

Compared to FP8, FP4 shows faster performance during the prefill stage, but FP4 is actually slightly slower during the decode stage.
According to profiling results, FP4Gemm is slower than (FP8)Gemm during decoding.
I'd like to ask whether this is due to a misconfiguration on my end or if the FP4 decode optimization is still in progress.

Additionally, is there a specific reason for applying 128-token dimension padding in Quantize_to_FP4 plugin?
For example, if the dimension of input activation at decode stage is [1x4096], the dimension of block-wise scale factor(FP8) should be [1x256] but actually it is [128x256] because of padding.
I'm wondering if this padding introduces any overhead during FP4Gemm.

Thank you.

Start with the examples/llm_ptq workflow and the linked cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.cpp padding logic. Reproduce the FP8 and NVFP4 prefill and decode benchmarks on an RTX 5090, then inspect the reported FP4Gemm profiling results. Done means identifying whether the decode slowdown or 128-token padding is expected, misconfigured, or needs an optimization.