NVIDIA / NVIDIA/cutlass

[BUG] cutlass-python does not work on H100 for CUDA 11.8

Open
#1,815 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage bug inactive-30d inactive-90d
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Describe the bug
The following sample program fails with the latest version of cutlass:

import cutlass
import numpy as np

plan = cutlass.op.Gemm(element=np.float16, layout=cutlass.LayoutType.RowMajor)
A, B, C, D = [np.ones((1024, 1024), dtype=np.float16) for i in range(4)]
plan.run(A, B, C, D)

The error given is as follows:

Error Message:
nvcc fatal   : Value 'sm_90a' is not defined for option 'gpu-architecture'

Steps/Code to reproduce bug

  • Download the latest version of cutlass on a H100 node that is configured with CUDA 11.8.
  • Build cutlass by following the installation instructions and (optionally) specify -DCUTLASS_NVCC_ARCHS="90"
  • Install the python library by running pip install . in the root directory of cutlass.
  • Run the example program.

Expected behavior
It looks like cutlass' python wrapper expects all H100 machines to want to run with CUDA 12+. I see, for instance, the following:

https://github.com/NVIDIA/cutlass/blob/3a8c01a18b24c35b216922481ac762496720a99d/python/cutlass/emit/pytorch.py#L691-L694

But that isn't necessarily the case.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with python/cutlass/emit/pytorch.py around lines 691-694 and reproduce the issue using the provided GEMM sample after building with CUDA 11.8 on an H100. Trace how the GPU architecture is selected, then verify that installing and running the Python example no longer produces the unsupported sm_90a error under CUDA 11.8.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
build-system, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.