NVIDIA / NVIDIA/cutlass

[BUG] Mixed Input H100 Kernel Hangs

Open
#2,121 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage bug inactive-30d inactive-90d
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

FE4M3 x BF16 Kernel Hangs when run with beta=1

Please compile the kernel cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma in profiler and run it with beta=0 and beta=1. It is not just the profiler, we are using this kernel for some of our shapes where it came to be the winner; however, it hangs when we run it with beta = 1, and our runs are not with profiler. So this might be a bug in the kernel and not just a profiler issue. We used CUDA 12.4.99.

beta = 1 hangs

./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma --m=128 --n=256 --k=512 --beta=1

beta = 0 works

./tools/profiler/cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma --m=128 --n=25
6 --k=512



=============================
  Problem ID: 1

        Provider: CUTLASS
   OperationKind: gemm
       Operation: cutlass3x_sm90_tensorop_s64x128x16gemm_e4m3_bf16_f32_bf16_bf16_cvt_64x128x128_8x1x1_0_tnt_align16_warpspecialized_pingpong_epi_tma

          Status: Success
    Verification: ON
     Disposition: Not verified

reference_device: Not run
          cuBLAS: Not run
           cuDNN: Not run

            Math: 4002.22 GFLOP/s
=============================

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/profiler/cutlass_profiler and reproduce the named CUTLASS kernel using the provided m=128, n=256, k=512 commands under CUDA 12.4.99, comparing beta=0 with beta=1. Determine whether the hang is in the kernel or profiler, and verify that the reported behavior is resolved or clearly characterized.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.