[QST] Performance gap between Debug mode and Release mode on GEMM kernel

Performance gap between Debug mode and Release mode

I try to use cutlass_profiler to profile GEMM kernel on H100 and analyze the source code of kernel.

cmake ..   -DCUTLASS_NVCC_ARCHS="90a"   -DCUTLASS_LIBRARY_KERNELS="cutlass3x_sm90_tensorop_gemm_f16_f16_f32_void_f16_128x128x64_8x1x1_0_tnn_align8_warpspecialized_pingpong_epi_tma"   -DCUTLASS_LIBRARY_INSTANTIATION_LEVEL="max"   -DCUTLASS_UNITY_BUILD_ENABLED=ON
make cutlass_profiler -j32
./cutlass_profiler --operation=gemm --m=16384 --n=16384 --k=512 --kernels=cutlass3x_sm90_tensorop_gemm_f16_f16_f32_void_f16_128x128x64_8x1x1_0_tnn_align8_warpspecialized_pingpong_epi_tma

Since release mode does not exhibit lineinfo of cuda code, I add -DCMAKE_BUILD_TYPE="Debug" to build with debug mode. But the performance downgrades a lot when I run cutlass_profiler again.

(with debug)
Runtime: 3.88052  ms
Memory: 8.05304 GiB/s
Math: 70973.6 GFLOP/s

(with release)
Runtime: 0.59216  ms
Memory: 52.7729 GiB/s
Math: 465102 GFLOP/s

Does anyone know why such a performance gap exists?

Start by comparing the Debug and Release CMake configurations and the reported cutlass_profiler commands for the H100 GEMM kernel. The investigation is complete when it explains the runtime difference and identifies a build configuration that provides the needed CUDA line information without the observed performance gap.