clab / clab/dynet

Error compiling Python 3 GPU support

Open
#1,570 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Hi, I'm trying to compile Dynet with GPU support with the following command `BACKEND=cuda pip install git+https://github.com/clab/dynet#egg=dynet` without success. This is the log of the error:

```Collecting dynet from git+https://github.com/clab/dynet#egg=dynet
Cloning https://github.com/clab/dynet to /tmp/pip-install-9jump_94/dynet
Running command git clone -q https://github.com/clab/dynet /tmp/pip-install-9jump_94/dynet
Requirement already satisfied: cython in ./miniconda3/envs/nlplab/lib/python3.7/site-packages (from dynet) (0.29.10)
Requirement already satisfied: numpy in ./miniconda3/envs/nlplab/lib/python3.7/site-packages (from dynet) (1.16.4)
Building wheels for collected packages: dynet
Building wheel for dynet (setup.py): started
Building wheel for dynet (setup.py): still running...
Building wheel for dynet (setup.py): still running...
Building wheel for dynet (setup.py): still running...
Building wheel for dynet (setup.py): still running...
Building wheel for dynet (setup.py): still running...
Building wheel for dynet (setup.py): still running...
Building wheel for dynet (setup.py): finished with status 'error'
ERROR: Complete output from command /home/user/miniconda3/envs/nlplab/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9jump_94/dynet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-t5wplbk9 --python-tag cp37:
ERROR: running bdist_wheel
running build
INFO:root:CMAKE_PATH='/usr/bin/cmake'
INFO:root:MAKE_PATH='/usr/bin/make'
INFO:root:MAKE_FLAGS='-j 4'
INFO:root:EIGEN3_INCLUDE_DIR='/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen'
INFO:root:EIGEN3_DOWNLOAD_URL='https://bitbucket.org/eigen/eigen/get/b2e267dc99d4.zip'
INFO:root:CC_PATH='/usr/bin/gcc'
INFO:root:CXX_PATH='/usr/bin/g++'
INFO:root:SCRIPT_DIR='/tmp/pip-install-9jump_94/dynet'
INFO:root:BUILD_DIR='/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit'
INFO:root:INSTALL_PREFIX='/home/user/miniconda3/envs/nlplab/lib/python3.7/site-packages/../../..'
INFO:root:PYTHON='/home/user/miniconda3/envs/nlplab/bin/python'
cmake version 3.14.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).
g++ (GCC) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

INFO:root:Creating build directory /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit
INFO:root:Fetching Eigen...
INFO:root:Unpacking Eigen...
INFO:root:Configuring...
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/g++
-- Check for working CXX compiler: /usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Optimization level: fast
-- BACKEND: cuda
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /opt/cuda (found version "10.1")
CUDA_LIBRARIES: /opt/cuda/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/librt.so;/opt/cuda/lib64/libcurand.so
-- Found CUDNN (include: /usr/include, library: /usr/lib/libcudnn.so)
-- Successfully include CUDNN flags
-- Eigen dir is /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen
-- Found Cython version 0.29.10

--- CUDA: CUBLAS: /opt/cuda/lib64/libcublas.so RT: /opt/cuda/lib64/libcudart_static.a;dl;/usr/lib/librt.so;/opt/cuda/lib64/libcurand.so;/usr/lib/libcudnn.so
CMAKE_INSTALL_PREFIX="/home/user/miniconda3/envs/nlplab"
PROJECT_SOURCE_DIR="/tmp/pip-install-9jump_94/dynet"
PROJECT_BINARY_DIR="/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit"
LIBS="/opt/cuda/lib64/libcudart_static.a\;dl\;/usr/lib/librt.so\;/opt/cuda/lib64/libcurand.so\;-lpthread"
EIGEN3_INCLUDE_DIR="/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen"
MKL_LINK_DIRS=""
WITH_CUDA_BACKEND="1"
CUDA_RT_FILES="libcudart_static.a\;dl\;librt.so\;libcurand.so\;libcudnn.so"
CUDA_RT_DIRS="/opt/cuda/lib64\;/usr/lib"
CUDA_CUBLAS_FILES="libcublas.so"
CUDA_CUBLAS_DIRS="/opt/cuda/lib64"
MSVC=""
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit
INFO:root:Compiling...
[ 1%] Building NVCC (Device) object dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-ops.cu.o
[ 2%] Building NVCC (Device) object dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge3.cu.o
[ 4%] Building NVCC (Device) object dynet/CMakeFiles/dynet.dir/dynet_generated_cudnn-ops.cu.o
[ 5%] Building NVCC (Device) object dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge0.cu.o
In file included from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/../../../Eigen/Core:269,
from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/Tensor:14,
from /tmp/pip-install-9jump_94/dynet/dynet/devices.h:14,
from /tmp/pip-install-9jump_94/dynet/dynet/cuda.h:16,
from /tmp/pip-install-9jump_94/dynet/dynet/gpu-ops.cu:1:
/opt/cuda/include/host_defines.h:54:2: warning: #warning "host_defines.h is an internal header file and must not be used directly. This file will be removed in a future CUDA release. Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
#warning "host_defines.h is an internal header file and must not be used directly. This file will be removed in a future CUDA release. Please use cuda_runtime_api.h or cuda_runtime.h instead."
^~~~~~~

[......]

/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/src/Tensor/TensorReduction.h(552): warning: calling a __host__ function("Eigen::internal::InnerReducer< ::Eigen::TensorEvaluator , const ::Eigen::array , const ::Eigen::TensorCwiseUnaryOp< ::Eigen::internal::bind2nd_op< ::Eigen::internal::scalar_product_op > , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_product_op , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_difference_op , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > > , ::Eigen::MakePointer> , ::Eigen::GpuDevice> , ::Eigen::internal::SumReducer , ::Eigen::GpuDevice> ::run ") from a __host__ __device__ function("Eigen::TensorEvaluator , const ::Eigen::array , const ::Eigen::TensorCwiseUnaryOp< ::Eigen::internal::bind2nd_op< ::Eigen::internal::scalar_product_op > , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_product_op , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_difference_op , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > > , ::Eigen::MakePointer> , ::Eigen::GpuDevice> ::evalSubExprsIfNeeded") is not allowed

/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/src/Tensor/TensorReduction.h(585): warning: calling a __host__ function("Eigen::internal::OuterReducer< ::Eigen::TensorEvaluator , const ::Eigen::array , const ::Eigen::TensorCwiseUnaryOp< ::Eigen::internal::bind2nd_op< ::Eigen::internal::scalar_product_op > , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_product_op , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_difference_op , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > > , ::Eigen::MakePointer> , ::Eigen::GpuDevice> , ::Eigen::internal::SumReducer , ::Eigen::GpuDevice> ::run") from a __host__ __device__ function("Eigen::TensorEvaluator , const ::Eigen::array , const ::Eigen::TensorCwiseUnaryOp< ::Eigen::internal::bind2nd_op< ::Eigen::internal::scalar_product_op > , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_product_op , const ::Eigen::TensorCwiseBinaryOp< ::Eigen::internal::scalar_difference_op , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > , const ::Eigen::TensorBroadcastingOp , const ::Eigen::TensorMap< ::Eigen::Tensor , (int)0, ::Eigen::MakePointer> > > > , ::Eigen::MakePointer> , ::Eigen::GpuDevice> ::evalSubExprsIfNeeded") is not allowed

In file included from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/../../../Eigen/Core:269,
from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/Tensor:14,
from /tmp/pip-install-9jump_94/dynet/dynet/devices.h:14,
from /tmp/pip-install-9jump_94/dynet/dynet/cuda.h:16,
from /tmp/pip-install-9jump_94/dynet/dynet/tensor.h:23,
from /tmp/pip-install-9jump_94/dynet/dynet/tensor-eigen.h:7,
from /tmp/pip-install-9jump_94/dynet/dynet/nodes-activations.cc:1,
from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/dynet/gpu-merge0.cu:1:
/opt/cuda/include/host_defines.h:54:2: warning: #warning "host_defines.h is an internal header file and must not be used directly. This file will be removed in a future CUDA release. Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
#warning "host_defines.h is an internal header file and must not be used directly. This file will be removed in a future CUDA release. Please use cuda_runtime_api.h or cuda_runtime.h instead."
^~~~~~~
In file included from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/../../../Eigen/Core:269,
from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen/unsupported/Eigen/CXX11/Tensor:14,
from /tmp/pip-install-9jump_94/dynet/dynet/devices.h:14,
from /tmp/pip-install-9jump_94/dynet/dynet/cuda.h:16,
from /tmp/pip-install-9jump_94/dynet/dynet/tensor.h:23,
from /tmp/pip-install-9jump_94/dynet/dynet/tensor-eigen.h:7,
from /tmp/pip-install-9jump_94/dynet/dynet/nodes-argmax.cc:1,
from /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/dynet/gpu-merge2.cu:1:
/opt/cuda/include/host_defines.h:54:2: warning: #warning "host_defines.h is an internal header file and must not be used directly. This file will be removed in a future CUDA release. Please use cuda_runtime_api.h or cuda_runtime.h instead." [-Wcpp]
#warning "host_defines.h is an internal header file and must not be used directly. This file will be removed in a future CUDA release. Please use cuda_runtime_api.h or cuda_runtime.h instead."
^~~~~~~
[ 6%] Building NVCC intermediate link file dynet/CMakeFiles/dynet.dir/dynet_intermediate_link.o
[ 8%] Linking CXX shared library libdynet.so
[ 98%] Built target dynet
[ 98%] Built target pydynet_precopy
[100%] Generating timestamp
running build_ext
INFO:root:Building Cython extensions...
INFO:root:INCLUDE_DIRS='/tmp/pip-install-9jump_94/dynet /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen'
INFO:root:LIBRARIES='dynet cublas'
INFO:root:LIBRARY_DIRS='/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/dynet/ . /opt/cuda/lib64'
INFO:root:COMPILER_ARGS='-std=c++11 -Wno-unused-function'
INFO:root:EXTRA_LINK_ARGS="-Wl,-rpath='/home/user/miniconda3/envs/nlplab/lib/',--no-as-needed"
INFO:root:RUNTIME_LIB_DIRS='/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/dynet/ /home/user/miniconda3/envs/nlplab/lib/'
cythoning /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.pyx to /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.cpp
building '_dynet' extension
/usr/bin/gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-9jump_94/dynet -I/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/eigen -I/home/user/miniconda3/envs/nlplab/include/python3.7m -c /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.cpp -o build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o -std=c++11 -Wno-unused-function
/usr/bin/g++ -pthread -shared -B /home/user/miniconda3/envs/nlplab/compiler_compat -L/home/user/miniconda3/envs/nlplab/lib -Wl,-rpath=/home/user/miniconda3/envs/nlplab/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o -L/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/dynet/ -L. -L/opt/cuda/lib64 -Wl,-R/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/dynet/ -Wl,-R/home/user/miniconda3/envs/nlplab/lib/ -ldynet -lcublas -o /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.cpython-37m-x86_64-linux-gnu.so -Wl,-rpath='/home/user/miniconda3/envs/nlplab/lib/',--no-as-needed
/home/user/miniconda3/envs/nlplab/compiler_compat/ld: build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o: unable to initialize decompress status for section .debug_info
/home/user/miniconda3/envs/nlplab/compiler_compat/ld: build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o: unable to initialize decompress status for section .debug_info
/home/user/miniconda3/envs/nlplab/compiler_compat/ld: build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o: unable to initialize decompress status for section .debug_info
/home/user/miniconda3/envs/nlplab/compiler_compat/ld: build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o: unable to initialize decompress status for section .debug_info
build/temp.linux-x86_64-3.7/tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
/home/user/miniconda3/envs/nlplab/lib/python3.7/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-9jump_94/dynet/build/py3.7-64bit/python/_dynet.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
error: command '/usr/bin/g++' failed with exit status 1
make[2]: *** [python/CMakeFiles/pydynet.dir/build.make:65: python/timestamp] Error 1
make[1]: *** [CMakeFiles/Makefile2:147: python/CMakeFiles/pydynet.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
error: /usr/bin/make -j 4
----------------------------------------
ERROR: Command "/home/user/miniconda3/envs/nlplab/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9jump_94/dynet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-upla1v9m/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-9jump_94/dynet/

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the BACKEND=cuda pip install command through setup.py and capture the complete compiler output beyond the truncated log. Inspect the CMake configuration and CUDA compilation failure; done means a Python 3.7 GPU-enabled installation completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, python
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.