Unable to Extract Throughput and Matrix Sizes for GEMM Kernels Using Nsight Systems Profiling in TensorRT-LLM

Description:

I am attempting to profile the performance of TensorRT-LLM during inference of Llama3-8b, specifically to gather throughput and matrix size data for each GEMM kernel. However, despite profiling the inference using Nsight Systems, I am unable to retrieve throughput information or matrix sizes for the GEMM kernels.

Steps Taken:

1. I used the following command to run the profiling with Nsight Systems, which tracks CUDA events, NVTX markers, and cuBLAS calls:

   nsys profile -w true --trace=cuda,nvtx,cublas --capture-range=cudaProfilerApi -o profresult python3 /path/to/your/script.py --max_output_len=50 --tokenizer_dir /path/to/tokenizer --engine_dir=/path/to/your/engine
   

2. During the profiling, I am able to see the time spent on individual kernels, but I am not able to find:

   • Throughput (operations per second) for each GEMM kernel.
   • Matrix sizes (input, weight, and output dimensions) involved in the GEMM operations.

3. I examined the output (profresult.nsys-rep) using Nsight Systems, but the matrix dimensions and throughput are not visible in the profiler report.

Questions:

1. How can I extract throughput data for individual GEMM kernels during inference in TensorRT-LLM?

   • Is there a specific configuration or flag in Nsight Systems or Nsight Compute that will capture throughput for GEMM operations?

2. How can I retrieve matrix sizes (input, weight, and output dimensions) for GEMM operations during inference?

   • Is there an API or method I can use to get the matrix dimensions related to GEMM kernels from TensorRT-LLM during inference?

3. Can I correlate kernel execution times with matrix dimensions to manually calculate throughput or determine the number of operations for GEMM kernels?