[BUG] BF16 * BF16 GEMM with pingpong schedule and non-zero beta hangs
Open
Nobody has claimed this yet.
? - Needs Triage
bug
inactive-30d
inactive-90d
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
cmake
cmake -DCMAKE_BUILD_TYPE:STRING=Release -DCUTLASS_NVCC_ARCHS:STRING=90a -DCUTLASS_NVCC_KEEP:STRING=OFF -DCUTLASS_ENABLE_F16C:STRING=ON -DCUTLASS_LIBRARY_KERNELS:STRING=cutlass3x_sm90_tensorop_s64x128x16gemm_bf16_bf16_f32_bf16_bf16_128x128x64*tnn_align8 -DCUTLASS_LIBRARY_INSTANTIATION_LEVEL:STRING=max -DCUTLASS_LIBRARY_IGNORE_KERNELS:STRING=gemm_grouped*,gemm_planar*,scl,sclzr -DCUTLASS_ENABLE_CUBLAS:STRING=ON -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -DCMAKE_C_COMPILER:FILEPATH=/usr/bin/gcc -DCMAKE_CXX_COMPILER:FILEPATH=/usr/bin/g++ --no-warn-unused-cli -S/home/manish_magic_dev/repos/cutlass/cutlass_tree_1/cutlass -B/home/manish_magic_dev/repos/cutlass/cutlass_tree_1/build
Profiler run command
beta=1, 2x2x1 cluster hangs
./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_bf16_bf16_f32_bf16_bf16_128x128x64_2x2x1_0_tnn_align8_warpspecialized_pingpong_epi_tma --beta=1
beta=0, 2x2x1 cluster does not hang
./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_bf16_bf16_f32_bf16_bf16_128x128x64_2x2x1_0_tnn_align8_warpspecialized_pingpong_epi_tma --beta=0
beta=1, 1x2x1 cluster hangs hangs
./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_bf16_bf16_f32_bf16_bf16_128x128x64_1x2x1_0_tnn_alig
n8_warpspecialized_pingpong_epi_tma --beta=1
beta=1, 1x1x1 cluster does not hang
./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_bf16_bf16_f32_bf16_bf16_128x128x64_1x1x1_0_tnn_align8_warpspecialized_pingpong_epi_tma --beta=1
Conclusions: Pingpong schedule with, non-zero beta, non-unity cluster shape (TMA-multicast) hangs.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cutlass_profiler entry point and reproduce the named BF16 GEMM kernel using the supplied beta and cluster-shape commands. Compare beta=1 against beta=0 and compare the 2x2x1 and 1x1x1 configurations to isolate the hang. Done means the pingpong schedule completes for non-zero beta with non-unity cluster shapes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- hpc, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100