NVIDIA / NVIDIA/cutlass

[BUG] CuTe DSL 4.8 dev regresses SM80 SDPA backward compilation (NVVM constraint 'n')

Open
#3,594 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage bug CuTe DSL
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Which component has the problem?

CuTe DSL

Bug Report

Describe the bug:
A cuDNN Frontend FROST SDPA backward case that compiles and runs
successfully with nvidia-cutlass-dsl 4.7.0 consistently fails during
NVVM compilation with 4.8.0 dev public wheel. This blocks the cuDNN
Frontend CUTLASS 4.8 SM80 FROST SDPA lane.

Steps to reproduce:

  1. Check out NVIDIA/cudnn-frontend at:
    4af0c9f645cf41a3547c638f9eec0c2861ff7aa3
  2. Install nvidia-cutlass-dsl[cu13]==4.8.0.dev0 from the 4.8 dev wheel channel.
  3. Run:

CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1
CUDNN_TEST_NO_ISOLATION=1
pytest -q -s
'test/python/test_mhas_v2.py::test_sdpa_random_bwd_L0[test468]'

Observed error:
NVVM_ERROR_COMPILATION:
constraint 'n' expects an integer constant expression

Expected behavior:
The case should compile and pass as it does with CuTe DSL 4.7.0.

Environment:

  • Bare metal NVIDIA A100-PCIE-40GB, sm80
  • CUDA 13.4
  • Driver 595.58.03
  • Python 3.12.3
  • PyTorch 2.14.0a0+4fdf77b940.nv26.08
  • cuDNN 9.27.0

Impact:
An FE-side workaround would require changing accumulator materialization
in a hot kernel and may increase register pressure. Please address this
compiler regression before CUTLASS 4.8 GA.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure from NVIDIA/cudnn-frontend commit 4af0c9f645cf41a3547c638f9eec0c2861ff7aa2 using test/python/test_mhas_v2.py::test_sdpa_random_bwd_L0[test468] and the stated environment. Compare compilation with nvidia-cutlass-dsl 4.7.0 and 4.8.0.dev0, then trace the NVVM constraint 'n' error in the CuTe DSL compilation path. Done means the SM80 FROST SDPA backward test compiles and passes on the 4.8 path without the FE workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, pytorch
Domain
compilers, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.