[BUG] Mixed Input H100 Kernel Hangs
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
FE4M3 x BF16 Kernel Hangs when run with beta=1
Please compile the kernel cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma in profiler and run it with beta=0 and beta=1. It is not just the profiler, we are using this kernel for some of our shapes where it came to be the winner; however, it hangs when we run it with beta = 1, and our runs are not with profiler. So this might be a bug in the kernel and not just a profiler issue. We used CUDA 12.4.99.
beta = 1 hangs
./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma --m=128 --n=256 --k=512 --beta=1
beta = 0 works
./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma --m=128 --n=25
6 --k=512
=============================
Problem ID: 1
Provider: CUTLASS
OperationKind: gemm
Operation: cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma
Status: Success
Verification: ON
Disposition: Not verified
reference_device: Not run
cuBLAS: Not run
cuDNN: Not run
Math: 4002.22 GFLOP/s
=============================
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tools/profiler/cutlass_profiler and reproduce the named CUTLASS kernel using the provided m=128, n=256, k=512 commands under CUDA 12.4.99, comparing beta=0 with beta=1. Determine whether the hang is in the kernel or profiler, and verify that the reported behavior is resolved or clearly characterized.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- hpc, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100