NVIDIA / NVIDIA/cutlass

[BUG] Example 93 GQA initialization overwrites active remote contributions

Open
#3,598 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Which component has the problem?

CUTLASS C++

Bug Report

Describe the bug

Example 93 initializes active split mailboxes after cluster startup synchronization. A receiving CTA can erase a contribution already written by a remote producer. Incorrect attention output was observed under Compute Sanitizer racecheck.

Steps/Code to reproduce bug

Use revision 246e9206: CUTLASS 59e3a3338d516ca6ce0e073af8da65289678a35c plus the separate scratch-reduction barrier. Keeping that barrier removes scratch reuse as a cause.

curl -fL https://raw.githubusercontent.com/1sgtpepper/cutlass/923e9400d6901f90f30d18510320d08d838fbcac/test/regression/gqa_max_scratch.cu -o gqa_max_scratch.cu
nvcc -std=c++17 -O3 -lineinfo --expt-relaxed-constexpr \
  -gencode=arch=compute_100a,code=sm_100a \
  -Iinclude -Itools/util/include -Iexamples/93_blackwell_low_latency_gqa \
  gqa_max_scratch.cu -o gqa_max_scratch
compute-sanitizer --tool racecheck --error-exitcode 3 \
  ./gqa_max_scratch --mailboxes --repetitions 3

Across seven eight-CTA cases, output contained 473 wrong values, despite zero reported racecheck hazards. Cases include all-active producers and inactive producers with a partial tail. The reference is double-precision softmax with 1/256 absolute tolerance.

Expected behavior

Active producers exclusively write their mailbox contributions. Inactive entries contain the neutral values needed by the final reduction.

Environment details (please complete the following information):

  • CUTLASS revisions above; main as of September 8, 2026: 59e3a3338d516ca6ce0e073af8da65289678a35c. The affected mailbox initialization is unchanged in v4.8.0dev and v4.7.1; these releases were not executed.
  • B200/SM100, driver 580.95.05.
  • CUDA 13.1.1 and bundled Compute Sanitizer.
  • Modal cloud GPU, Ubuntu 24.04 container, Linux x86_64.

Additional context

The local epilog barrier cannot order remote stores against initialization. Initializing only inactive entries changes the wrong-value count from 473 to 0; restoring the old initialization gives 196. Counts vary with scheduling. Ordinary launches passed; these numerical failures occurred under racecheck.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported command with test/regression/gqa_max_scratch.cu, the referenced CUTLASS revision, and Compute Sanitizer racecheck. Inspect mailbox initialization in examples/93_blackwell_low_latency_gqa and compare initialization of active and inactive entries. Done means the reproduced cases produce no wrong values while inactive entries retain the neutral values required by the final reduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.