NVIDIA / NVIDIA/cutlass

[BUG] Example 93 GQA reduction races when reusing shared scratch

Open
#3,597 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Which component has the problem?

CUTLASS C++

Bug Report

Describe the bug

Example 93 can overwrite shared maximum-reduction scratch before all four warps finish reading it. Under Compute Sanitizer racecheck, this produces shared-memory hazards and incorrect attention output.

Steps/Code to reproduce bug

In a checkout of 59e3a3338d516ca6ce0e073af8da65289678a35c:

curl -fL https://raw.githubusercontent.com/1sgtpepper/cutlass/923e9400d6901f90f30d18510320d08d838fbcac/test/regression/gqa_max_scratch.cu -o gqa_max_scratch.cu
nvcc -std=c++17 -O3 -lineinfo --expt-relaxed-constexpr \
  -gencode=arch=compute_100a,code=sm_100a \
  -Iinclude -Itools/util/include -Iexamples/93_blackwell_low_latency_gqa \
  gqa_max_scratch.cu -o gqa_max_scratch
compute-sanitizer --tool racecheck --error-exitcode 3 \
  ./gqa_max_scratch --repetitions 3

The fixture calls the production kernel with two CTAs, 512 KV tokens, head dimension 64 and three stages. A double-precision softmax reference checks BF16 output with 1/256 absolute tolerance. One run produced 2,193 wrong values and eight race hazards; counts vary with scheduling.

Expected behavior

Every participating warp finishes reading a reduction's scratch before another reduction reuses it.

Environment details (please complete the following information):

  • CUTLASS 59e3a3338d516ca6ce0e073af8da65289678a35c, main as of September 8, 2026. The affected reduction is unchanged in v4.8.0dev and v4.7.1; these releases were not executed.
  • B200/SM100, driver 580.95.05; CUDA 13.1.1 and bundled Compute Sanitizer.
  • Modal cloud GPU, Ubuntu 24.04 container, Linux x86_64.

Additional context

cta_reduce synchronizes before reading warp partials but not after reading them. Its caller reuses the same scratch for subsequent KV tiles. Ordinary launches passed; the numerical failures above were observed under racecheck.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the production kernel's cta_reduce path in Example 93 and the gqa_max_scratch.cu fixture linked in the report. Build the fixture with nvcc, then run it under Compute Sanitizer racecheck to reproduce the shared-memory hazards. Done means the fixture reports no race hazards and its BF16 output passes the stated reference tolerance.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.