NVIDIA / NVIDIA/cutlass

[QST] Cutlass Profiler

Open
#2,362 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage inactive-30d inactive-90d question
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

What is your question?
When I run

cutlass_profiler --operation=gemm --enable-best-kernel-for-fixed-shape --m=6144 --n=6144 --k=6144 --sort-results-flops-per-sec --output=gemm.csv

The cluster_m, cluster_n columns are always 1, even for gemms with procedural names like:

cutlass3x_sm90_tensorop_gemm_f16_f16_f32_f16_f16_256x128x64_1x2x1_0_tnn_align8_warpspecialized_cooperative_epi_tma

which have a static ClusterShape (1x2x1 in the given example) encoded in the CollectiveMainloop.

However, when I run the following for the same set of compiled gemms:

cutlass_profiler --operation=gemm  --m=6144 --n=6144 --k=6144 --output=gemm.no_search.csv

the cluster_m and cluster_n columns match those of the template.

Using cutlass 4.0 where I compiled only a subset of gemms matching the pattern cutlass3x_sm90_tensorop_gemm_f16_f16_f32_f16_f16_*_tn*.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the two cutlass_profiler invocations and compare the --enable-best-kernel-for-fixed-shape path with the ordinary GEMM reporting path. Reproduce the discrepancy using the stated SM90 procedural kernel pattern and fixed 6144x6144x6144 shape; done means cluster_m and cluster_n consistently reflect the compiled template in both outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.