NVIDIA / NVIDIA/cutlass

[BUG] Unexpected SystemExit: nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(911): diagnostic

Open
#3,142 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage bug CuTe DSL inactive-30d inactive-90d
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Which component has the problem?

CuTe DSL

Bug Report

Describe the bug

Executing NVBench Python example script using CuteDSL runs into "error: argument -diagnostic: invalid choice: '0' (choose from all, fail, success, info, suggestion)"

Screenshot of reported behavior
$ python cute_dsl_sgemm.py -d 0 -a "R=64" -a "N=256" --stopping-criterion entropy
# Devices

## [0] `NVIDIA RTX A6000`
* SM Version: 860 (PTX Version: 750)
* Number of SMs: 84
* SM Default Clock Rate: 1800 MHz
* Global Memory: 43478 MiB Free / 48532 MiB Total
* Global Memory Bus Peak: 768 GB/sec (384-bit DDR @8001MHz)
* Max Shared Memory: 100 KiB/SM, 48 KiB/Block
* L2 Cache Size: 6144 KiB
* Maximum Active Blocks: 16/SM
* Maximum Active Threads: 1536/SM, 1024/Block
* Available Registers: 65536/SM, 65536/Block
* ECC Enabled: No

# Log

Run: [1/1] cutlass_gemm [Device=0 R=64 N=256]
usage: cute_dsl_sgemm.py [-h] [-diagnostic [{all,fail,success,info,suggestion}]]
cute_dsl_sgemm.py: error: argument -diagnostic: invalid choice: '0' (choose from all, fail, success, info, suggestion)
Fail: Unexpected error: SystemExit: 2

At:
/home/opavlyk/miniforge/envs/evo/lib/python3.12/argparse.py(2637): exit
/home/opavlyk/miniforge/envs/evo/lib/python3.12/argparse.py(2650): error
/home/opavlyk/miniforge/envs/evo/lib/python3.12/argparse.py(1945): _parse_known_args2
/home/opavlyk/miniforge/envs/evo/lib/python3.12/argparse.py(1914): parse_known_args
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(911): diagnostic
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(998): compile_and_jit
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(1229): compile_and_cache
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/cutlass_dsl/cutlass.py(557): compile_and_cache
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(1407): generate_mlir
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(1567): _func
/home/opavlyk/miniforge/envs/evo/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py(457): jit_wrapper
/home/opavlyk/repos/nvbench/python/examples/cute_dsl_sgemm.py(636): cutlass_gemm
/home/opavlyk/repos/nvbench/python/examples/cute_dsl_sgemm.py(650):


# Benchmark Results

## cutlass_gemm

### [0] NVIDIA RTX A6000

No data -- check log.
Segmentation fault

Steps/Code to reproduce bug

On a machine with CTK 13 installed, and GPU available:

mkdir -p /tmp/reproducer
pushd /tmp/reproducer
git clone https://github.com/oleksandr-pavlyk/nvbench -b update-python-examples
cd nvbench/python

conda create -n cute-dsl-bug --yes python=3.12 pip 
conda activate cute-dsl-bug

# install dependencies to build NVBench python package
pip install cython scikit-build-core setuptools-scm typing-extensions cmake cuda-bindings

export CUDACXX=/usr/local/cuda/bin/nvcc
python -m pip install --no-build-isolation --no-deps --verbose .

cd examples

# install example dependencies
pip install cuda-core cupy nvidia-cutlass-dsl[cu13] 

# run the example that reproduces the behavior
python cute_dsl_sgemm.py -d 0 -a R=64 -a N=1024

I am aware of a work-around, which is to modify $CONDA_PREFIX/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py using the following diff:

$ diff $CONDA_PREFIX/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl_orig.py $CONDA_PREFIX/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py
911c911
<         args, _ = parser.parse_known_args()
---
>         # args, _ = parser.parse_known_args()
920c920
<         if args.diagnostic is None:
---
>         if True or args.diagnostic is None:

Expected behavior

The example should run without errors, like after applying the work-around

Expected script output
(report-bug) opavlyk@ee09c48-lcedt:/tmp/opavlyk/nvbench/python/examples$ python cute_dsl_sgemm.py -d 0 -a R=64 -a N=1024 --profile
# Devices

## [0] `NVIDIA RTX A6000`
* SM Version: 860 (PTX Version: 750)
* Number of SMs: 84
* SM Default Clock Rate: 1800 MHz
* Global Memory: 43627 MiB Free / 48532 MiB Total
* Global Memory Bus Peak: 768 GB/sec (384-bit DDR @8001MHz)
* Max Shared Memory: 100 KiB/SM, 48 KiB/Block
* L2 Cache Size: 6144 KiB
* Maximum Active Blocks: 16/SM
* Maximum Active Threads: 1536/SM, 1024/Block
* Available Registers: 65536/SM, 65536/Block
* ECC Enabled: No

# Log

Run: [1/1] cutlass_gemm [Device=0 R=64 N=1024]
Pass: Cold: 89.330688ms GPU, 89.342430ms CPU, 0.09s total GPU, 0.09s total wall, 1x


# Benchmark Results

## cutlass_gemm

### [0] NVIDIA RTX A6000

| R  |  N   | Samples | CPU Time  | Noise | GPU Time  | Noise |
|----|------|---------|-----------|-------|-----------|-------|
| 64 | 1024 |      1x | 89.342 ms |  inf% | 89.331 ms |  inf% |

Environment details (please complete the following information):

Bare metal, Ubuntu 24.04, CTK 13.1,

Wed Apr  1 09:15:36 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.44.01              Driver Version: 590.44.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               On  |   00000000:01:00.0 Off |                  Off |
| 39%   63C    P2             87W /  300W |    4712MiB /  49140MiB |      6%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A400                On  |   00000000:C1:00.0  On |                  N/A |
| 30%   42C    P8            N/A  /   50W |     131MiB /   4094MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Additional context

In the same environment:

cd ../../../
git clone https://github.com/NVIDIA/cutlass
cd cutlass/examples/python/CuTeDSL/ampere

# install missing dependency
pip install torch[cu13]

# run the original example (runs fine)
python sgemm.py

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/dsl.py around the diagnostic function and compare its argument parsing with nvbench/python/examples/cute_dsl_sgemm.py. Reproduce using the provided CTK 13 environment and command, then verify that the example completes with benchmark data instead of SystemExit and a segmentation fault.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.