error when building torchaudio v0.9.0 from source on ppc64le (power9)
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 798
- Avg merge
- 58m
- Merged PRs (30d)
- 3
Description
🐛 Describe the bug
Hi there,
I am trying to build torchaudio v0.9.0 with pytorch 1.9 from source (looks like this post is trying to do the same thing), on a power9 machine. However, I am running into a cmake error when trying to build (below is error snippit; will provide full error at the end of this post):
...
#$ gcc -D__CUDA_ARCH__=750 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS
-D__CUDACC__ -D__NVCC__
"-I/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/..//include"
-D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=2
-D__CUDACC_VER_BUILD__=89 -include "cuda_runtime.h"
"CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.compute_75.cpp1.ii"
#$ cicc --c++14 --gnu_version=80301 --allow_managed --unsigned_chars -arch
compute_75 -m64 -ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 --include_file_name
"CMakeCUDACompilerId.fatbin.c" -tused -nvvmir-library
"/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/../nvvm/libdevice/libdevice.10.bc"
--gen_module_id_file --module_id_file_name
"tmp/CMakeCUDACompilerId.module_id" --orig_src_file_name
"CMakeCUDACompilerId.cu" --gen_c_file_name
"tmp/CMakeCUDACompilerId.compute_75.cudafe1.c" --stub_file_name
"tmp/CMakeCUDACompilerId.compute_75.cudafe1.stub.c" --gen_device_file_name
"tmp/CMakeCUDACompilerId.compute_75.cudafe1.gpu"
"tmp/CMakeCUDACompilerId.compute_75.cpp1.ii" -o
"tmp/CMakeCUDACompilerId.compute_75.ptx"
/usr/include/c++/8/type_traits(335): error: identifier "__ieee128" is
undefined
/usr/include/bits/floatn.h(79): error: identifier "__ieee128" is undefined
/usr/include/bits/floatn.h(82): error: invalid argument to attribute
"__mode__"
Things I have done so far:
- install pytorch 1.9 and cudadevtools on a conda kernel with pytorch 3.7
- pip install cmake and ninja (previous errors indicated I should do this)
- modified
third_party/CMakeLists.txt' to include flags per [this](https://github.com/LLNL/blt/issues/341) issue, i.e.set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -Xcompiler -mno-float128")` - run
python setup.py installwithMAX_JOBS=1andBUILD_SOX=1
and I am getting the following full error below:
(torch1.9new) [jzhao7@node0009 audio]$ python setup.py install
-- Git branch: main
-- Git SHA: b986e9ef304e0c5fa1f4e82164a29a51b66ea498
-- Git tag: None
-- PyTorch dependency: torch
-- Building version 0.11.0a0+b986e9e
--- Initializing submodules
--- Initialized submodule
running install
running bdist_egg
running egg_info
writing torchaudio.egg-info/PKG-INFO
writing dependency_links to torchaudio.egg-info/dependency_links.txt
writing requirements to torchaudio.egg-info/requires.txt
writing top-level names to torchaudio.egg-info/top_level.txt
reading manifest file 'torchaudio.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'torchaudio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-ppc64le/egg
running install_lib
running build_py
copying torchaudio/version.py -> build/lib.linux-ppc64le-3.7/torchaudio
running build_ext
CMake Error at /nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:724 (message):
Compiling the CUDA compiler identification source file
"CMakeCUDACompilerId.cu" failed.
Compiler: /nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/nvcc
Build flags:
Id flags:
--keep;--keep-dir;tmp;-gencode=arch=compute_37,code=sm_37;-gencode=arch=compute_60,code=sm_60;-gencode=arch=compute_70,code=sm_70;-gencode=arch=compute_75,code=sm_75
-v
The output was:
1
#$ _NVVM_BRANCH_=nvvm
#$ _SPACE_=
#$ _CUDART_=cudart
#$ _HERE_=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin
#$ _THERE_=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_SIZE_=64
#$ TOP=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/..
#$
NVVMIR_LIBRARY_DIR=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/../nvvm/libdevice
#$
LD_LIBRARY_PATH=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/../lib:::/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/extras/CUPTI/lib64::/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/cuda/lib:
#$
LD_LIBRARY_PATH=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/../lib:::/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/extras/CUPTI/lib64::/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/cuda/lib:
#$
PATH=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/../nvvm/bin:/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin:/home/jzhao7/.local/bin:/home/jzhao7/bin:/home/jzhao7/.cargo/bin:/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin:/
nobackup/users/jzhao7/anaconda3/condabin:/home/jzhao7/.local/bin:/home/jzhao7/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
#$
INCLUDES="-I/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/..//include"
#$ LIBRARIES=
"-L/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/..//lib64/stubs"
"-L/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/..//lib64"
#$ CUDAFE_FLAGS=
#$ PTXAS_FLAGS=
#$ rm tmp/a_dlink.reg.c
#$ gcc -D__CUDA_ARCH__=750 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS
-D__CUDACC__ -D__NVCC__
"-I/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/..//include"
-D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=2
-D__CUDACC_VER_BUILD__=89 -include "cuda_runtime.h"
"CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.compute_75.cpp1.ii"
#$ cicc --c++14 --gnu_version=80301 --allow_managed --unsigned_chars -arch
compute_75 -m64 -ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 --include_file_name
"CMakeCUDACompilerId.fatbin.c" -tused -nvvmir-library
"/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/bin/../nvvm/libdevice/libdevice.10.bc"
--gen_module_id_file --module_id_file_name
"tmp/CMakeCUDACompilerId.module_id" --orig_src_file_name
"CMakeCUDACompilerId.cu" --gen_c_file_name
"tmp/CMakeCUDACompilerId.compute_75.cudafe1.c" --stub_file_name
"tmp/CMakeCUDACompilerId.compute_75.cudafe1.stub.c" --gen_device_file_name
"tmp/CMakeCUDACompilerId.compute_75.cudafe1.gpu"
"tmp/CMakeCUDACompilerId.compute_75.cpp1.ii" -o
"tmp/CMakeCUDACompilerId.compute_75.ptx"
/usr/include/c++/8/type_traits(335): error: identifier "__ieee128" is
undefined
/usr/include/bits/floatn.h(79): error: identifier "__ieee128" is undefined
/usr/include/bits/floatn.h(82): error: invalid argument to attribute
"__mode__"
/usr/include/c++/8/bits/std_abs.h(101): error: identifier "__ieee128" is
undefined
/usr/include/c++/8/bits/std_abs.h(102): error: identifier "__ieee128" is
undefined
5 errors detected in the compilation of
"tmp/CMakeCUDACompilerId.compute_75.cpp1.ii".
# --error 0x1 --
Call Stack (most recent call first):
/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
CMakeLists.txt:85 (enable_language)
-- Configuring incomplete, errors occurred!
See also "/home/jzhao7/audionew/audio/build/temp.linux-ppc64le-3.7/CMakeFiles/CMakeOutput.log".
See also "/home/jzhao7/audionew/audio/build/temp.linux-ppc64le-3.7/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "setup.py", line 181, in <module>
_main()
File "setup.py", line 176, in _main
zip_safe=False,
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/command/install.py", line 109, in do_egg_install
self.run_command('bdist_egg')
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 164, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 150, in call_command
self.run_command(cmdname)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/command/install_lib.py", line 11, in run
self.build()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/command/install_lib.py", line 107, in build
self.run_command('build_ext')
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/jzhao7/audionew/audio/tools/setup_helpers/extension.py", line 71, in run
super().run()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/home/jzhao7/audionew/audio/tools/setup_helpers/extension.py", line 141, in build_extension
subprocess.check_call(["cmake", str(_ROOT_DIR)] + cmake_args, cwd=self.build_temp)
File "/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/jzhao7/audionew/audio', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/lib/python3.7/site-packages/torch/share/cmake', '-DCMAKE_INSTALL_PREFIX=/home/jzhao7/audionew/audio/build/lib.li
nux-ppc64le-3.7/torchaudio/', '-DCMAKE_VERBOSE_MAKEFILE=ON', '-DPython_INCLUDE_DIR=/nobackup/users/jzhao7/anaconda3/envs/torch1.9new/include/python3.7m', '-DBUILD_SOX:BOOL=ON', '-DBUILD_FFMPEG:BOOL=OFF', '-DBUILD_KALDI:BOOL=ON', '-DBUILD_RNNT:BOOL=ON', '-DBUILD_CTC_DECODER:BOOL=ON', '-DBUILD_T
ORCHAUDIO_PYTHON_EXTENSION:BOOL=ON', '-DUSE_ROCM:BOOL=OFF', '-DUSE_CUDA:BOOL=ON', '-DUSE_OPENMP:BOOL=ON', '-DCMAKE_CUDA_ARCHITECTURES=37-real;60-real;70-real;75-real', '-GNinja']' returned non-zero exit status 1.
Thanks so much for your help in advance!
Versions
Collecting environment information...
PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A
OS: Red Hat Enterprise Linux release 8.3 (Ootpa) (ppc64le)
GCC version: (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Clang version: Could not collect
CMake version: version 3.22.2
Libc version: glibc-2.17
Python version: 3.7.11 (default, Jul 27 2021, 15:02:35) [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-240.el8.ppc64le-ppc64le-with-redhat-8.3-Ootpa
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
Nvidia driver version: 460.73.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.9.0
[pip3] torchtext==0.10.0a0+4da1de3
[pip3] torchvision==0.10.0
[conda] _pytorch_select 2.0 cuda_2 https://opence.mit.edu
[conda] cudatoolkit 10.2.89 698.g82312a1 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access
[conda] cudatoolkit-dev 10.2.89 698.g82312a1 https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access
[conda] numpy 1.19.2 py37h6163131_0
[conda] numpy-base 1.19.2 py37h75fe3a5_0
[conda] pytorch 1.9.0 cuda10.2_py37_1 https://opence.mit.edu
[conda] pytorch-base 1.9.0 h1234567_cuda10.2_py37_pb3.14_1 https://opence.mit.edu
[conda] torchtext 0.10.0 py37_1 https://opence.mit.edu
[conda] torchvision-base 0.10.0 cuda10.2_py37_1 https://opence.mit.edu
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with setup.py and third_party/CMakeLists.txt, then inspect CMake's compiler-identification failure and the reported nvcc command on ppc64le. Reproduce with the stated PyTorch 1.9, CUDA, CMake, and Python setup, and compare the existing -mno-float128 workaround. Done means the torchaudio source build completes successfully on the Power9 environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, python, pytorch
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100