Nvidia Modelopt structured 2:4 weight sparsity showing no speed improvement compared to dense model

Describe the bug

A Qwen3-4B-Instruct-2507 hugging face model sparsified to 2:4 structured sparsity[Weight sparsity] with ModelOpt
(SparseGPT post-training sparsification + sparsity-aware fine-tuning) shows ~0% inference
speedup versus the equivalent dense model when deployed with TensorRT-LLM, despite the
weights being verified as genuine, hardware-valid 2:4.

Verification of the sparse checkpoint:

• GEMM weights: 50.00% zeros (dense baseline: 0.00%)
• LayerNorm / embedding weights correctly untouched (0% zeros)

TensorRT-LLM engine is built after PTS+SAT -> TRTLLM ckpt export:

1. --gemm_plugin auto --weight_sparsity — GEMMs routed to cuBLASLt (dense path).

Impact: should-have. The full sparsification cost (pruning + fine-tune + build)
is paid, but yields no throughput/latency benefit and no memory-footprint reduction on this
configuration.

Open question for maintainers: is any speedup expected for FP16 2:4 in a decode-bound /
small-batch regime on Ada (L4), and does the stock trtllm-build path actually
dispatch sparse (cuSPARSELt / native sparse) GEMM kernels for Qwen3 linear layers? If not,
what precision (FP8 / INT8), batch regime, and GPU architecture are required for ModelOpt 2:4
sparsity to show a measurable gain?

Steps/Code to reproduce bug

1. Sparsify Qwen3-4B-Instruct-2507 to 2:4 with ModelOpt (SparseGPT + sparsity-aware
   fine-tuning) and export to a TensorRT-LLM checkpoint.(Used process in https://github.com/NVIDIA/Model-Optimizer/blob/main/examples/llm_sparsity/weight_sparsity)

2. Verify the checkpoint is truly 2:4 (50% zeros in GEMM weights; every 4-group has exactly
   2 zeros on the last dim; norms untouched).

3. Build the sparse engine:

   trtllm-build \
     --checkpoint_dir <sparse_ckpt> \
     --output_dir <sparse_engine> \
     --gemm_plugin auto \
     --weight_sparsity \
     --profiling_verbosity detailed \
     --max_batch_size 64 --max_input_len 4096 --max_seq_len 8192 --max_num_tokens 32768
   

4. Build a dense engine identically, but without --weight_sparsity.

5. Serve each engine (trtllm-serve, --backend tensorrt, --tp_size 1, port 8000).

6. Benchmark both with guidellm, fixed shape prompt_tokens=512, output_tokens=256,
   --profile concurrent --rate 1,2,4,8,16,24,32,64,128 --max-requests 200 --max-seconds 120.

Result: sparse ≈ dense on all metrics; both plateau at ~485 gen tok/s around concurrency
~32; ITL identical (~33.8 ms). No speedup from 2:4.

Expected behavior

Either:

• a measurable GEMM / throughput speedup from 2:4 structured sparsity on Sparse Tensor Cores, or
• If it don't work on Ada Lovelace GPUS, Will it work on A10G GPU? or only A100

Who can help?

System information

• Container used (if applicable): TensorRT-LLM container (build + serve); ModelOpt installed via pip
• OS (e.g., Ubuntu 22.04): Ubuntu 24.04.4 LTS
• CPU architecture (x86_64, aarch64): x86_64
• GPU name (e.g. H100, A100, L40S): NVIDIA L4 (Ada Lovelace, sm_89, AWS g6.xlarge)
• GPU memory size: 24 GB (23034 MiB)
• Number of GPUs: 1
• Library versions (if applicable):
  • Python: 3.12.3
  • ModelOpt version or commit hash: 0.45.0
  • CUDA: 13.2 (driver / nvidia-smi); PyTorch built against CUDA 13.1; NVIDIA driver 595.58.03
  • PyTorch: 2.11.0a0+eb65b36914.nv26.02
  • Transformers: 5.5.4
  • TensorRT-LLM: 1.3.0rc18
  • ONNXRuntime: not installed (N/A)
  • TensorRT: 10.15.1.29
• Any other details that may help:
  • Model: Qwen3-4B-Instruct-2507, 2:4 structured-sparse, FP16.
  • Sparsity verified real (50% zeros; last-dim 2:4-valid = 1.0; norms untouched).
  • Two build variants tested (--gemm_plugin auto and --gemm_plugin disable --multiple_profiles enable); both give ~0% vs dense.
  • Footprint not reduced by 2:4 (non-zeros stored in FP16 + metadata); the --gemm_plugin disable engine is ~12 GB vs ~8.9 GB dense.
  • Workload is decode-bound / small-to-moderate batch, which may explain the lack of gain; seeking confirmation whether this is expected rather than a defect.

##Help
I already performed quantization techniques. And I want to know if performing sparsity along with quantization worth it? because I didn't see any performance speedup on sparsity alone.