NVIDIA / NVIDIA/apex

bf16 support for FusedDense preventing apex build on CUDA 10.2

Open
#1,670 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
9k
Forks
1.5k
Avg merge
2d 4h
Merged PRs (30d)
3

Description

Describe the Bug

Minimal Steps/Code to Reproduce the Bug

bf16 support for FusedDense is added in https://github.com/NVIDIA/apex/pull/1627

04207351f5e5491546f09bae169cf76d1cd820d9 is successfully built on 10.2: https://github.com/minostauros/pytorch-video-docker/actions/runs/4732576445/jobs/8398928476
8b7a1ff183741dd8f9b87e7bafd04cfde99cea28 is not: https://github.com/minostauros/pytorch-video-docker/actions/runs/5070398012/jobs/9105381516#step:7:1824

log: 9_docker (pytorch-1.9.0torchvision-0.10.0cuda-10.2ffmpeg-4.2.txt

#6 987.9   /usr/local/cuda/bin/nvcc  -I/usr/local/lib/python3.7/dist-packages/torch/include -I/usr/local/lib/python3.7/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.7/dist-packages/torch/include/TH -I/usr/local/lib/python3.7/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.7m -c -c /tmp/apex/csrc/fused_dense_cuda.cu -o /tmp/apex/build/temp.linux-x86_64-cpython-37/csrc/fused_dense_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=fused_dense_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_37,code=sm_37 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14
#6 987.9   /tmp/apex/csrc/fused_dense_cuda.cu(156): error: identifier "CUDA_R_16BF" is undefined

Evironment

ARG PYTHON_VERSION=3.7
ARG PYTORCH_VERSION=1.9.0
ARG TORCHVISION_VERSION=0.10.0
ARG TORCHAUDIO_VERSION=0.9.0
ARG TARGET_CUDA_VERSION=cu102
ARG CUDA_VERSION=10.2

# Kepler(partial) to Turing (3.5-7.5) with CUDA 10.2
ENV TORCH_CUDA_ARCH_LIST=3.5;3.7;5.0;5.2;6.0;6.1;6.2;7.0;7.5

RUN DIR=/tmp/apex && \
    mkdir -p ${DIR} && \
    cd ${DIR} && \
    git clone https://github.com/minostauros/apex . && \
    pip${PYTHON_VERSION} install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./ && \
    rm -rf ${DIR} && \
    cd /tmp/

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with csrc/fused_dense_cuda.cu, especially the reported error near line 156, and compare the bf16 support introduced by NVIDIA/apex#1627 with the CUDA 10.2 build command and log. Reproduce the failure using the provided Python 3.7, PyTorch 1.9.0, and CUDA 10.2 environment; done means the Apex CUDA extension builds successfully on that setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.