NVIDIA / NVIDIA/cutlass

[BUG] Int8 multiplication with pytorch extension: namespace "torch" has no member "I8

Open
#1,573 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage bug inactive-30d inactive-90d
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Describe the bug
To compile cutlass code to python torch:I8 is used which does not exist. The int8 is named torch:k8I in pytorch

Steps/Code to reproduce bug

import torch
import cutlass

plan = cutlass.op.Gemm(
    element=cutlass.DataType.s8,
    element_accumulator=cutlass.DataType.s32,
    element_D=cutlass.DataType.s32,
    layout=cutlass.LayoutType.RowMajor)
op = plan.construct()

mod = cutlass.emit.pytorch(op, name='gemm_mod', cc=plan.cc, jit=True)

a = torch.empty((2048, 4096), dtype=torch.int8, device=0)
b = torch.empty((4096, 8192), dtype=torch.int8, device=0)

D_ref = a.float() @ b.float()
D = mod.run(a, b)

print(D)
print(D_ref)
Traceback (most recent call last):
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2107, in _run_ninja_build
    subprocess.run(
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "benchmark/main.py", line 22, in <module>
    
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass/emit/pytorch.py", line 927, in pytorch
    return _pytorch_gemm(device_op, name, cc, jit, sourcedir)
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass/emit/pytorch.py", line 775, in _pytorch_gemm
    return _jit(name, cc, cpp_file, cuda_file)
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass/emit/pytorch.py", line 697, in _jit
    jitmodule = load(
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1309, in load
    return _jit_compile(
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1719, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1832, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2123, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'gemm_mod': [1/3] c++ -MMD -MF gemm_mod.o.d -DTORCH_EXTENSION_NAME=gemm_mod -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass_library/source/include -I/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass_library/source/tools/util/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/TH -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/THC -isystem /home/mcelik/anaconda3/envs/cublas8bit/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /home/mcelik/Projects/torch_cublas_matmul_int8/gemm_mod.cpp -o gemm_mod.o 
[2/3] /home/mcelik/anaconda3/envs/cublas8bit/bin/nvcc --generate-dependencies-with-compile --dependency-output gemm_mod_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=gemm_mod -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass_library/source/include -I/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass_library/source/tools/util/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/TH -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/THC -isystem /home/mcelik/anaconda3/envs/cublas8bit/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -std=c++17 -c /home/mcelik/Projects/torch_cublas_matmul_int8/gemm_mod_kernel.cu -o gemm_mod_kernel.cuda.o 
FAILED: gemm_mod_kernel.cuda.o 
/home/mcelik/anaconda3/envs/cublas8bit/bin/nvcc --generate-dependencies-with-compile --dependency-output gemm_mod_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=gemm_mod -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass_library/source/include -I/home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/cutlass_library/source/tools/util/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/TH -isystem /home/mcelik/anaconda3/envs/cublas8bit/lib/python3.8/site-packages/torch/include/THC -isystem /home/mcelik/anaconda3/envs/cublas8bit/include -isystem /home/mcelik/anaconda3/envs/cublas8bit/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -std=c++17 -c /home/mcelik/Projects/torch_cublas_matmul_int8/gemm_mod_kernel.cu -o gemm_mod_kernel.cuda.o 
/home/mcelik/Projects/torch_cublas_matmul_int8/gemm_mod_kernel.cu(106): error: namespace "torch" has no member "I8"
      at::Tensor D = B.new_empty({M, N}, torch::I8);
                                                ^

1 error detected in the compilation of "/home/mcelik/Projects/torch_cublas_matmul_int8/gemm_mod_kernel.cu".
ninja: build stopped: subcommand failed.

Expected behavior
Multiplication performed as expected.

Environment details (please complete the following information):
CUDA A6000 GPU
Pytorch 2.3.1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Inspect the PyTorch emitter referenced by cutlass/emit/pytorch.py and compare it with the generated gemm_mod_kernel.cu at the failing output line. Reproduce the issue with the provided cutlass.op.Gemm and emit.pytorch example, then verify that the generated extension compiles and the int8 multiplication runs as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.