jupyter / jupyter/notebook

Notebook cuda version does not follow LD_LIBRARY_PATH

Open
#5,871 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
13.3k
Forks
5.8k
Avg merge
6d 11h
Merged PRs (30d)
7

Description

I have setup my `.profile` as follows:

```
if [ -d "/usr/local/cuda-10.1/bin/" ]; then   
export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}   
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}fi
fi
```

so I correctly load in the shell the right `nvcc` version:

```
$ nvcc -Vnvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
```

and I can see GPU devices

```python
>>>
from tensorflow.python.client import device_lib
def get_available_gpus():...    
local_device_protos = device_lib.list_local_devices()...    
return [x.name for x in local_device_protos]... 
>>> get_available_gpus()
['/device:CPU:0', '/device:XLA_CPU:0', '/device:XLA_GPU:0', '/device:GPU:0']
```

and run Tensorflow against the right CUDA version:

```
>>> import tensorflow as tf
2020-11-17 19:54:13.688704: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
>>> assert tf.test.is_gpu_available()
Created TensorFlow device (/device:GPU:0 with 7113 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
```

But when I start the notebook I get a different older version of cuda:

Schermata 2020-11-17 alle 20 07 52

so I will get an error if I try to access the GPU device 0.

I have tried to set the `env` variable in the `kernel.json` file but I'm not sure if it works in this way:

```json
$ jupyter kernelspec list
Available kernels:
python3 /usr/local/share/jupyter/kernels/python3
$ cat /usr/local/share/jupyter/kernels/python3/kernel.json
{
"argv": [
"/usr/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python",
"env": {"LD_LIBRARY_PATH":"/usr/local/cuda-10.1/lib64"}
}
```

I run the notebook as follows:

```
jupyter notebook --no-browser --allow-root --port=$PORT --ip='*' --NotebookApp.token='' --NotebookApp.password='' 2>&1 &
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the mismatch with the shown `jupyter notebook` command, then inspect `/usr/local/share/jupyter/kernels/python3/kernel.json` and compare the notebook kernel environment with the interactive shell. Done means the notebook kernel consistently uses the CUDA 10.1 library path and can access the GPU without the reported version error.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, shell
Domain
developer-experience, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.