[QST]How to optimize the tensorrt-llm's mixed gemm

What is your question?
Recently ,i used the tensorrt-llm's example to run the qwen 1.5-2.5B inference by A100 GPU.According to ncu's analysis report.Some gemm can be optimised. In that report said that the grid for this launch is configured to execute only 32 blocks, which is less than the GPU's 108 multiprocessors. This can underutilize some multiprocessors. I have tried to use smaller threadblock shape,but it had no effect.

For example tensorrt-llm used threadblock shape and warp shape is <16,128,64> <16,32,64> ,and i tried to use threadblock shape and warp shape are <8,128,64> <8,64,64>.

Working directory/tensorrt_llm/examples/models/core/qwen .
1、Download the Qwen2.5-1.5B(-Instruct).
2、transform the data:python: convert_checkpoint.py --model_dir ./Qwen2.5-1.5B-Instruct --output_dir ./output_dir --dtype float16 --use_weight_only --weight_only_precision int8
3、compile:trtllm-build --checkpoint_dir ./output_dir --output_dir ./1-gpu --gemm_plugin float16 > trt-build.txt.
4、python3 ../../../run.py --input_text "熵"
--max_output_len=10
--tokenizer_dir Qwen2.5-1.5B-Instruct
--engine_dir=./1-gpu/

In fpA_intB_gemm_template.h file,i added some code to print kernel's run time and this gemm's shape.

In that way ,i got ncu's report:
ncu -o profile --target-processes all -f python3 ../../../run.py --input_text "熵" --max_output_len=1 --tokenizer_dir ./tmp/Qwen/Qwen2.5-1.5B-Instruct/ --engine_dir=./tmp/qwen/Qwen2.5-1.5B-Instruct/trt_engines/w8a8/1-gpu/ --cuda_graph_mode --no_prompt_template 2>&1 | tee log.txt
mv profile.ncu-rep qwen2.5-1.5B-16b-w8a8.ncu-rep
ncu --import qwen2.5-1.5B-16b-w8a8.ncu-rep > ncu-qwen2.5-1.5B-16b-w8a8.txt

Start in tensorrt_llm/examples/models/core/qwen and inspect the reported GEMM path in fpA_intB_gemm_template.h. Reproduce the Qwen2.5-1.5B build and inference commands, then collect the Nsight Compute report described in the issue. Done means identifying and validating a concrete optimization for the underutilized mixed GEMM.