error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
When I tried to run
python3 setup.py install --cuda_ext --cpp_ext
I got
` torch.version = 1.2.0
setup.py:46: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
from /usr/local/cuda/bin
running install
running bdist_egg
running egg_info
writing apex.egg-info/PKG-INFO
writing dependency_links to apex.egg-info/dependency_links.txt
writing top-level names to apex.egg-info/top_level.txt
reading manifest file 'apex.egg-info/SOURCES.txt'
writing manifest file 'apex.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/apex
copying apex/init.py -> build/lib.linux-x86_64-3.7/apex
creating build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/init.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN
creating build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/init.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/version.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/initialize.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp
creating build/lib.linux-x86_64-3.7/apex/contrib
copying apex/contrib/init.py -> build/lib.linux-x86_64-3.7/apex/contrib
creating build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/init.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/init.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
creating build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/init.py -> build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization
creating build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/init.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_lamb.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_novograd.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/optimizers
creating build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/init.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
creating build/lib.linux-x86_64-3.7/apex/pyprof
copying apex/pyprof/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof
creating build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/init.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
creating build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/init.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
creating build/lib.linux-x86_64-3.7/apex/contrib/groupbn
copying apex/contrib/groupbn/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn
copying apex/contrib/groupbn/batch_norm.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn
creating build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
creating build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
creating build/lib.linux-x86_64-3.7/apex/contrib/xentropy
copying apex/contrib/xentropy/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy
copying apex/contrib/xentropy/softmax_xentropy.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy
creating build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
copying apex/pyprof/nvtx/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
copying apex/pyprof/nvtx/nvmarker.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
creating build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/main.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/db.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/kernel.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/nvvp.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/parse.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
creating build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/main.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/activation.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/base.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/blas.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/conv.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/convert.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/data.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/dropout.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/embedding.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/index_slice_join_mutate.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/linear.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/loss.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/misc.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/normalization.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/optim.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/output.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/pointwise.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/pooling.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/prof.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/randomSample.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/recurrentCell.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/reduction.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/softmax.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/usage.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/utility.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
running build_ext
building 'apex_C' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/csrc
/home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include -fPIC -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/narimene/anaconda3/include/python3.7m -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-c++ -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/narimene/anaconda3/lib -Wl,-rpath-link,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so
building 'amp_C' extension
/home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include -fPIC -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/narimene/anaconda3/include/python3.7m -c csrc/amp_C_frontend.cpp -o build/temp.linux-x86_64-3.7/csrc/amp_C_frontend.o -O3 -DVERSION_GE_1_1 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/usr/local/cuda/bin/nvcc -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/narimene/anaconda3/include/python3.7m -c csrc/multi_tensor_sgd_kernel.cu -o build/temp.linux-x86_64-3.7/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/core/TensorTypeIdRegistration.h(50): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/typeid.h(596): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/typeid.h(596): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/Registry.h(157): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(23): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(24): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(26): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(30): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(34): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/Generator.h(75): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/ATenDispatch.h(104): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(114): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(115): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(69): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(75): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(207): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(218): error: namespace "std" has no member "mutex"
17 errors detected in the compilation of "/tmp/tmpxft_0000624a_00000000-6_multi_tensor_sgd_kernel.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1 `
Please if someone can help me
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with setup.py and rerun python3 setup.py install --cuda_ext --cpp_ext, capturing the complete compiler output after the reported nvcc failure. Review the build involving csrc/flatten_unflatten.cpp and the CUDA/PyTorch versions shown in the report; done means the extension installation completes successfully or the incompatibility is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100