NVIDIA / NVIDIA/cutlass

[QST] Variable size Gemm that can be Cuda graphed

Open
#2,164 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage inactive-30d inactive-90d question
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

It is common with LLMs to have variable sized inputs, which usually means that the GEMMs cannot be used with a cuda graph.
However, there is a specific case for which a solution is possible. When Computing A * B, the common dimension of the two matrices does not change the launch configuration. This means that there could be an early exit from the loop that traverses the tiles in a single strip.
If the size is stored on device (rather than passed through the kernel API), this means that the kernel can be used with cuda graph as well.
My question is:

  • Does an existing Cutlass functionality already expose this varying size behavior that can be graphed?
  • Is there a way to add this "early exit" with a minimal intervention into the code? Is there a relevant example?

This is super useful for low latency attention computation with small batch...

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name a file, test, or entry point. Start by reviewing existing CUTLASS GEMM and CUDA graph functionality, then determine whether device-stored varying dimensions already work. Done means documenting an applicable existing mechanism or defining and validating a minimal early-exit approach, with a relevant example if one exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.