[BUG] Example 93 GQA initialization overwrites active remote contributions
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Which component has the problem?
CUTLASS C++
Bug Report
Describe the bug
Example 93 initializes active split mailboxes after cluster startup synchronization. A receiving CTA can erase a contribution already written by a remote producer. Incorrect attention output was observed under Compute Sanitizer racecheck.
Steps/Code to reproduce bug
Use revision 246e9206: CUTLASS 59e3a3338d516ca6ce0e073af8da65289678a35c plus the separate scratch-reduction barrier. Keeping that barrier removes scratch reuse as a cause.
curl -fL https://raw.githubusercontent.com/1sgtpepper/cutlass/923e9400d6901f90f30d18510320d08d838fbcac/test/regression/gqa_max_scratch.cu -o gqa_max_scratch.cu
nvcc -std=c++17 -O3 -lineinfo --expt-relaxed-constexpr \
-gencode=arch=compute_100a,code=sm_100a \
-Iinclude -Itools/util/include -Iexamples/93_blackwell_low_latency_gqa \
gqa_max_scratch.cu -o gqa_max_scratch
compute-sanitizer --tool racecheck --error-exitcode 3 \
./gqa_max_scratch --mailboxes --repetitions 3
Across seven eight-CTA cases, output contained 473 wrong values, despite zero reported racecheck hazards. Cases include all-active producers and inactive producers with a partial tail. The reference is double-precision softmax with 1/256 absolute tolerance.
Expected behavior
Active producers exclusively write their mailbox contributions. Inactive entries contain the neutral values needed by the final reduction.
Environment details (please complete the following information):
- CUTLASS revisions above; main as of September 8, 2026:
59e3a3338d516ca6ce0e073af8da65289678a35c. The affected mailbox initialization is unchanged inv4.8.0devandv4.7.1; these releases were not executed. - B200/SM100, driver 580.95.05.
- CUDA 13.1.1 and bundled Compute Sanitizer.
- Modal cloud GPU, Ubuntu 24.04 container, Linux x86_64.
Additional context
The local epilog barrier cannot order remote stores against initialization. Initializing only inactive entries changes the wrong-value count from 473 to 0; restoring the old initialization gives 196. Counts vary with scheduling. Ordinary launches passed; these numerical failures occurred under racecheck.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported command with test/regression/gqa_max_scratch.cu, the referenced CUTLASS revision, and Compute Sanitizer racecheck. Inspect mailbox initialization in examples/93_blackwell_low_latency_gqa and compare initialization of active and inactive entries. Done means the reproduced cases produce no wrong values while inactive entries retain the neutral values required by the final reduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100