[BUG] Example 93 GQA reduction races when reusing shared scratch
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Which component has the problem?
CUTLASS C++
Bug Report
Describe the bug
Example 93 can overwrite shared maximum-reduction scratch before all four warps finish reading it. Under Compute Sanitizer racecheck, this produces shared-memory hazards and incorrect attention output.
Steps/Code to reproduce bug
In a checkout of 59e3a3338d516ca6ce0e073af8da65289678a35c:
curl -fL https://raw.githubusercontent.com/1sgtpepper/cutlass/923e9400d6901f90f30d18510320d08d838fbcac/test/regression/gqa_max_scratch.cu -o gqa_max_scratch.cu
nvcc -std=c++17 -O3 -lineinfo --expt-relaxed-constexpr \
-gencode=arch=compute_100a,code=sm_100a \
-Iinclude -Itools/util/include -Iexamples/93_blackwell_low_latency_gqa \
gqa_max_scratch.cu -o gqa_max_scratch
compute-sanitizer --tool racecheck --error-exitcode 3 \
./gqa_max_scratch --repetitions 3
The fixture calls the production kernel with two CTAs, 512 KV tokens, head dimension 64 and three stages. A double-precision softmax reference checks BF16 output with 1/256 absolute tolerance. One run produced 2,193 wrong values and eight race hazards; counts vary with scheduling.
Expected behavior
Every participating warp finishes reading a reduction's scratch before another reduction reuses it.
Environment details (please complete the following information):
- CUTLASS
59e3a3338d516ca6ce0e073af8da65289678a35c, main as of September 8, 2026. The affected reduction is unchanged inv4.8.0devandv4.7.1; these releases were not executed. - B200/SM100, driver 580.95.05; CUDA 13.1.1 and bundled Compute Sanitizer.
- Modal cloud GPU, Ubuntu 24.04 container, Linux x86_64.
Additional context
cta_reduce synchronizes before reading warp partials but not after reading them. Its caller reuses the same scratch for subsequent KV tiles. Ordinary launches passed; the numerical failures above were observed under racecheck.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the production kernel's cta_reduce path in Example 93 and the gqa_max_scratch.cu fixture linked in the report. Build the fixture with nvcc, then run it under Compute Sanitizer racecheck to reproduce the shared-memory hazards. Done means the fixture reports no race hazards and its BF16 output passes the stated reference tolerance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100