NVIDIA / NVIDIA/apex

error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1

Open
#797 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.5k
Avg merge
2d 4h
Merged PRs (30d)
3

Description

When I tried to run
python3 setup.py install --cuda_ext --cpp_ext

I got
` torch.version = 1.2.0

setup.py:46: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
from /usr/local/cuda/bin

running install
running bdist_egg
running egg_info
writing apex.egg-info/PKG-INFO
writing dependency_links to apex.egg-info/dependency_links.txt
writing top-level names to apex.egg-info/top_level.txt
reading manifest file 'apex.egg-info/SOURCES.txt'
writing manifest file 'apex.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/apex
copying apex/init.py -> build/lib.linux-x86_64-3.7/apex
creating build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/init.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN
creating build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/init.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/version.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/initialize.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp
creating build/lib.linux-x86_64-3.7/apex/contrib
copying apex/contrib/init.py -> build/lib.linux-x86_64-3.7/apex/contrib
creating build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/init.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/init.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
creating build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/init.py -> build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization
creating build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/init.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_lamb.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_novograd.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/optimizers
creating build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/init.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
creating build/lib.linux-x86_64-3.7/apex/pyprof
copying apex/pyprof/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof
creating build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/init.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
creating build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/init.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
creating build/lib.linux-x86_64-3.7/apex/contrib/groupbn
copying apex/contrib/groupbn/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn
copying apex/contrib/groupbn/batch_norm.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn
creating build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
creating build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
creating build/lib.linux-x86_64-3.7/apex/contrib/xentropy
copying apex/contrib/xentropy/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy
copying apex/contrib/xentropy/softmax_xentropy.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy
creating build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
copying apex/pyprof/nvtx/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
copying apex/pyprof/nvtx/nvmarker.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
creating build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/main.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/db.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/kernel.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/nvvp.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/parse.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
creating build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/main.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/activation.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/base.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/blas.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/conv.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/convert.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/data.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/dropout.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/embedding.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/index_slice_join_mutate.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/linear.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/loss.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/misc.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/normalization.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/optim.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/output.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/pointwise.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/pooling.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/prof.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/randomSample.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/recurrentCell.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/reduction.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/softmax.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/usage.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/utility.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
running build_ext
building 'apex_C' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/csrc
/home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include -fPIC -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/narimene/anaconda3/include/python3.7m -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-c++ -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/narimene/anaconda3/lib -Wl,-rpath-link,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so
building 'amp_C' extension
/home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include -fPIC -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/narimene/anaconda3/include/python3.7m -c csrc/amp_C_frontend.cpp -o build/temp.linux-x86_64-3.7/csrc/amp_C_frontend.o -O3 -DVERSION_GE_1_1 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/usr/local/cuda/bin/nvcc -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/narimene/anaconda3/include/python3.7m -c csrc/multi_tensor_sgd_kernel.cu -o build/temp.linux-x86_64-3.7/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS
-D__CUDA_NO_HALF_CONVERSIONS
_ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/core/TensorTypeIdRegistration.h(50): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/typeid.h(596): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/typeid.h(596): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/Registry.h(157): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(23): error: namespace "std" has no member "once_flag"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(24): error: namespace "std" has no member "once_flag"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(26): error: namespace "std" has no member "call_once"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(30): error: namespace "std" has no member "call_once"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(34): error: namespace "std" has no member "call_once"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/Generator.h(75): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/ATenDispatch.h(104): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(114): error: namespace "std" has no member "once_flag"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(115): error: namespace "std" has no member "once_flag"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(69): error: namespace "std" has no member "call_once"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(75): error: namespace "std" has no member "call_once"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(207): error: namespace "std" has no member "mutex"

/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(218): error: namespace "std" has no member "mutex"

17 errors detected in the compilation of "/tmp/tmpxft_0000624a_00000000-6_multi_tensor_sgd_kernel.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1 `

Please if someone can help me

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with setup.py and rerun python3 setup.py install --cuda_ext --cpp_ext, capturing the complete compiler output after the reported nvcc failure. Review the build involving csrc/flatten_unflatten.cpp and the CUDA/PyTorch versions shown in the report; done means the extension installation completes successfully or the incompatibility is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.