Notebook cuda version does not follow LD_LIBRARY_PATH
- Dominant language
- Jupyter Notebook
- Stars
- 13.3k
- Forks
- 5.8k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 7
Description
I have setup my `.profile` as follows:
```
if [ -d "/usr/local/cuda-10.1/bin/" ]; then
export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}fi
fi
```
so I correctly load in the shell the right `nvcc` version:
```
$ nvcc -Vnvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
```
and I can see GPU devices
```python
>>>
from tensorflow.python.client import device_lib
def get_available_gpus():...
local_device_protos = device_lib.list_local_devices()...
return [x.name for x in local_device_protos]...
>>> get_available_gpus()
['/device:CPU:0', '/device:XLA_CPU:0', '/device:XLA_GPU:0', '/device:GPU:0']
```
and run Tensorflow against the right CUDA version:
```
>>> import tensorflow as tf
2020-11-17 19:54:13.688704: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> assert tf.test.is_gpu_available()
Created TensorFlow device (/device:GPU:0 with 7113 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
```
But when I start the notebook I get a different older version of cuda:

so I will get an error if I try to access the GPU device 0.
I have tried to set the `env` variable in the `kernel.json` file but I'm not sure if it works in this way:
```json
$ jupyter kernelspec list
Available kernels:
python3 /usr/local/share/jupyter/kernels/python3
$ cat /usr/local/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"/usr/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python",
"env": {"LD_LIBRARY_PATH":"/usr/local/cuda-10.1/lib64"}
}
```
I run the notebook as follows:
```
jupyter notebook --no-browser --allow-root --port=$PORT --ip='*' --NotebookApp.token='' --NotebookApp.password='' 2>&1 &
```
Contributor guide
Research direction
Start by reproducing the mismatch with the shown `jupyter notebook` command, then inspect `/usr/local/share/jupyter/kernels/python3/kernel.json` and compare the notebook kernel environment with the interactive shell. Done means the notebook kernel consistently uses the CUDA 10.1 library path and can access the GPU without the reported version error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, shell
- Domain
- developer-experience, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100