NVIDIA / NVIDIA/apex

error: command '/usr/bin/nvcc' failed with exit status 1

Open
#1,012 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.5k
Avg merge
2d 4h
Merged PRs (30d)
3

Description

Hello,

When running apex command python setup.py install --cuda_ext --cpp_ext I get the following error error: command '/usr/bin/nvcc' failed with exit status 1

Here's my system details:

cudatoolkit: 10.0.130
Pytorch: 1.4.0
Python: 3.6.9
GCC: 7.3.0
$which gcc
/usr/bin/gcc
$which nvcc
/usr/bin/nvcc
$nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85
$which g++
/usr/bin/g++
$which make
/usr/bin/make
vendor   : NVIDIA Corporation
model    : GP107M [GeForce GTX 1050 Mobile]
driver   : nvidia-driver-450
$nvidia-smi
 NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0  

I also should mention that I was having this error raised when I was trying to install apex:

"Cuda extensions are being compiled with a version of Cuda that does " +
                           "not match the version used to compile Pytorch binaries.  " +
                           "Pytorch binaries were compiled with Cuda {}.\n".format(torch.version.cuda) +
                           "In some cases, a minor-version mismatch will not cause later errors:  " +
                           "https://github.com/NVIDIA/apex/pull/323#discussion_r287021798.  "
                           "You can try commenting out this check (at your own risk)."

So I just commented the check to avoid this.
if (bare_metal_major != torch_binary_major) or (bare_metal_minor != torch_binary_minor):

I see that maybe there are some conflicts somewhere: nvidia-smi shows cuda version of 11.0, nvcc -V gives 9.1 and I am using 10.0. Maybe I got the wrong driver ?
Also I see that the path of nvcc is /usr/bin/nvcc instead of /usr/cuda/bin/nvcc

Any ideas on the source of this error error: command '/usr/bin/nvcc' failed with exit status 1 ?

Thank you

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported python setup.py install --cuda_ext --cpp_ext command and the CUDA-version check shown in the issue. Compare the reported PyTorch, toolkit, nvcc, driver, GCC, and CUDA versions, then capture the compiler output beyond the exit status; done means identifying the specific compatibility or build failure and documenting a reproducible resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.