clab / clab/dynet

CMake Error: could not generate ./dynet_generated_gpu-merge2.cu.o

Open
#1,613 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

I followed the instructions outlined in Dynet docs. These are my setups.

CMake Command
cmake .. -DEIGEN3_INCLUDE_DIR=/usr/local/include/eigen -DENABLE_CPP_EXAMPLES=ON -DBACKEND=cuda -DCUDNN_ROOT=/usr/local/cuda/lib64 -DENABLE_BOOST=ON -DPYTHON=`which python`

```
Dependencies
g++ version
g++ (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0

CUDA Version
/usr/local/cuda-10.1/

CUDNN Version
7.6.5

Boost Version
Version: 1.65.1.0ubuntu1

Python Version
/home/briansia/anaconda3/bin/python
Python 3.7.3

Eigen Version
3.3.90
```

Running the CMake produces the following output.
```
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 8.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Optimization level: fast
-- Enabling Boost
-- Found Boost: /usr/include (found version "1.65.1") found components: program_options regex serialization
-- Boost dir is /usr/include
-- BACKEND: cuda
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "10.1")
CUDA_LIBRARIES: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so;/usr/local/cuda/lib64/libcurand.so
-- Found CUDNN (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Successfully include CUDNN flags
-- Eigen dir is /usr/local/include/eigen
-- Found Cython version 0.29.15

--- CUDA: CUBLAS: /usr/lib/x86_64-linux-gnu/libcublas.so RT: /usr/local/cuda/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib/x86_64-linux-gnu/librt.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcudnn.so
CMAKE_INSTALL_PREFIX="/usr/local"
PROJECT_SOURCE_DIR="/home/briansia/dynet"
PROJECT_BINARY_DIR="/home/briansia/dynet/build"
LIBS="/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.65.1\;/usr/lib/x86_64-linux-gnu/libboost_regex.so.1.65.1\;/usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.65.1\;/usr/local/cuda/lib64/libcudart_static.a\;Threads::Threads\;dl\;/usr/lib/x86_64-linux-gnu/librt.so\;/usr/local/cuda/lib64/libcurand.so\;-lpthread"
EIGEN3_INCLUDE_DIR="/usr/local/include/eigen"
MKL_LINK_DIRS=""
WITH_CUDA_BACKEND="1"
CUDA_RT_FILES="libcudart_static.a\;Threads::Threads\;dl\;librt.so\;libcurand.so\;libcudnn.so"
CUDA_RT_DIRS="/usr/local/cuda/lib64\;/usr/lib/x86_64-linux-gnu"
CUDA_CUBLAS_FILES="libcublas.so"
CUDA_CUBLAS_DIRS="/usr/lib/x86_64-linux-gnu"
MSVC=""
-- Found Boost: /usr/include (found version "1.65.1") found components: system filesystem unit_test_framework
-- Configuring done
-- Generating done
-- Build files have been written to: /home/briansia/dynet/build
```
I could not find out why CMAKE_HAVE_LIBC_PTHREAD failed and pthread_create could not be found in the CUDA backend. However, I proceeded to run make anyways.

Here is the output from make -j 12.

[make.log](https://github.com/clab/dynet/files/4776068/make.log)

The very end of the log showed that make failed because of the following lines.

```
dynet/CMakeFiles/dynet.dir/build.make:97: recipe for target 'dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge0.cu.o' failed

[lots of ptxas error thrown here]

Error generating file
/home/briansia/dynet/build/dynet/CMakeFiles/dynet.dir//./dynet_generated_gpu-merge2.cu.o
```
Does anyone have an idea as to what caused this? Thanks.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with make.log and the CMake-generated target dynet/CMakeFiles/dynet.dir/dynet_generated_gpu-merge2.cu.o, comparing the failing ptxas output with the CUDA 10.1, cuDNN 7.6.5, Eigen, Boost, and compiler settings in the issue. Re-run the shown CMake command and make invocation to isolate the configuration or compilation cause; done means the CUDA backend builds without the ptxas error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.