jupyter / jupyter/notebook

Nginx + notebook issue

Open
#625 27 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
13.3k
Forks
5.8k
Avg merge
6d 11h
Merged PRs (30d)
7

Description

Hi guys,

I've been trying to get my jupyter notebook to run behind an nginx reverse proxy and everything seems to be going well, until i actually try to connect to the kernel in a notebook. Here's my `/etc/nginx/conf.d/configuration.conf` configuration upfront:

``` bash
upstream notebook {
server localhost:8888;
}

server {
listen 443;
ssl on;
server_name notebook.mydomain.com www.notebook.mydomain.com;
ssl_certificate /home//certificates/ipythoncert.pem;
ssl_certificate_key /home//certificates/ipythoncert.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
access_log /home//.jupyter/log/nginx.access.log;
error_log /home//.jupyter/log/nginx.error.log;

location / {
proxy_pass https://notebook;
proxy_set_header Host $host;
}

location /api/kernels/ {
proxy_pass https://notebook;
proxy_set_header Host $host;
# websocket support
proxy_http_version 1.1;
proxy_set_header Upgrade "websocket";
proxy_set_header Connection "Upgrade";
proxy_read_timeout 86400;
}
}
```

Also, here's my `~/.jupyter/jupyter_notebook_config.py`:

``` python
c.NotebookApp.ip = 'localhost'
c.NotebookApp.allow_origin = '*'
c.NotebookApp.certfile = u'/home//certificates/ipythoncert.pem'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:imnotthatstupiddontworry'
c.NotebookApp.trust_xheaders = True
#everything else is left commented
```

On the client, the `

...
` won't stop displaying "connecting to kernel", and I can't get my cells to run.

As it retries, I get a popup with "A connection to the notebook server could not be established. The notebook will continue trying to reconnect, but until it does, you will NOT be able to run code. Check your network connection or notebook server configuration."

and, the javascript console:

``` javascript
> Default extension for cell metadata editing loaded.
> Raw Cell Format toolbar preset loaded.
> Slideshow extension for metadata editing loaded.
> https://notebook.mydomain.com/nbextensions/widgets/notebook/js/extension.js Failed to load resource: the server responded with a status of 404 (Not Found)
> ipywidgets package not installed. Widgets are not available.
> Session: kernel_created (5e294362-a965-46c0-9bef-cfd052f91f9e)
> Starting WebSockets: wss://notebook.mydomain.com/api/kernels/a7fdec59-190e-405b-bd2f-35126d0b8981
> WebSocket connection to 'wss://notebook.mydomain.com/api/kernels/a7fdec59-190e-405b-bd2f-35126d0b8981/channels?session_id=' failed: Error during WebSocket handshake: Unexpected response code: 504
> Kernel: kernel_disconnected (a7fdec59-190e-405b-bd2f-35126d0b8981)
> WebSocket connection failed: wss://notebook.mydomain.com/api/kernels/
> Connection lost, reconnecting in 1 seconds.
> Kernel: kernel_reconnecting (a7fdec59-190e-405b-bd2f-35126d0b8981)
...
more failures
...
```

(Edit: I just noticed that in the above javascript console output, the id following the `kernel_created` message is different from all the other ids used in e.g. the kernel URLs; my guess is that isn't relevant at all, but I'm running out of ideas...)

from the server's perspective (loglevel "DEBUG"):

```
[D 21:48:20.451 NotebookApp] Using contents: services/contents
[D 21:48:20.452 NotebookApp] 200 GET /notebooks/Odd%20jobs.ipynb () 1.86ms
[D 21:48:20.574 NotebookApp] 200 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=9b2c8d3489227115310662a343fce11c () 0.82ms
[D 21:48:20.583 NotebookApp] 200 GET /static/components/MathJax/MathJax.js?config=TeX-AMS_HTML-full,Safe&delayStartupUntil=configured () 0.77ms
[D 21:48:20.588 NotebookApp] 200 GET /static/components/bootstrap-tour/build/css/bootstrap-tour.min.css?v=d0b3c2fce6056a2ddd5a4513762a94c4 () 0.69ms
[D 21:48:20.594 NotebookApp] 200 GET /static/components/codemirror/lib/codemirror.css?v=549e7432c1e8e94d1a2e66d7be92a430 () 0.63ms
[D 21:48:20.596 NotebookApp] 200 GET /static/style/style.min.css?v=726bda814675157da88304dabb766757 () 1.27ms
[D 21:48:20.610 NotebookApp] 200 GET /static/notebook/css/override.css?v=e6f18013b8771987812e992b38ec3318 () 0.75ms
[D 21:48:20.622 NotebookApp] 200 GET /custom/custom.css () 0.75ms
[D 21:48:20.635 NotebookApp] 200 GET /static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe () 0.55ms
[D 21:48:20.637 NotebookApp] 200 GET /static/components/requirejs/require.js?v=2de44fdcc1fe5e939aa4ce80626b241d () 0.78ms
[D 21:48:20.651 NotebookApp] 200 GET /static/components/text-encoding/lib/encoding.js?v=d5bb0fc9ffeff7d98a69bb83daa51052 () 0.87ms
[D 21:48:20.665 NotebookApp] 200 GET /static/notebook/js/main.min.js?v=40e10638fcf65fc1c057bff31d165e9d () 5.20ms
[D 21:48:20.680 NotebookApp] 200 GET /static/base/images/logo.png?v=7c4597ba713d804995e8f8dad448a397 () 0.70ms
[D 21:48:20.784 NotebookApp] 200 GET /static/components/MathJax/config/TeX-AMS_HTML-full.js?rev=2.5.3 () 1.57ms
[D 21:48:20.914 NotebookApp] 200 GET /static/style/style.min.css.map () 1.13ms
[D 21:48:21.387 NotebookApp] 200 GET /static/components/MathJax/config/Safe.js?rev=2.5.3 () 0.68ms
[D 21:48:21.877 NotebookApp] 200 GET /static/notebook/js/main.min.js.map () 5.87ms
[D 21:48:22.066 NotebookApp] 200 GET /static/services/contents.js?v=20151020214716 () 1.13ms
[D 21:48:22.080 NotebookApp] 200 GET /custom/custom.js?v=20151020214716 () 0.74ms
[D 21:48:22.132 NotebookApp] 200 GET /api/config/notebook?_=1445377700749 () 0.74ms
[D 21:48:22.135 NotebookApp] 200 GET /api/config/common?_=1445377700750 () 0.54ms
[D 21:48:22.491 NotebookApp] Native kernel (python2) available from /home//.virtualenvs//lib/python2.7/site-packages/ipykernel/resources
[D 21:48:22.491 NotebookApp] Native kernel (python2) available from /home//.virtualenvs//lib/python2.7/site-packages/ipykernel/resources
[D 21:48:22.491 NotebookApp] 200 GET /api/kernelspecs () 1.36ms
[D 21:48:22.496 NotebookApp] 200 GET /static/components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 () 0.82ms
[D 21:48:22.541 NotebookApp] 200 GET /api/contents/Odd%20jobs.ipynb?type=notebook&_=1445377700751 () 9.91ms
[D 21:48:22.671 NotebookApp] Using contents: services/contents
[W 21:48:22.672 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20151020214716 () 1.60ms referer=https://notebook.mydomain.com/notebooks/Odd%20jobs.ipynb
[D 21:48:22.729 NotebookApp] 200 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?rev=2.5.3 () 0.72ms
[D 21:48:23.067 NotebookApp] 201 POST /api/sessions () 0.86ms
[D 21:48:23.070 NotebookApp] 200 GET /api/contents/Odd%20jobs.ipynb/checkpoints?_=1445377700752 () 0.70ms
[D 21:48:23.089 NotebookApp] Native kernel (python2) available from /home//.virtualenvs//lib/python2.7/site-packages/ipykernel/resources
[D 21:48:23.089 NotebookApp] Serving kernel resource from: /home//.virtualenvs//lib/python2.7/site-packages/ipykernel/resources
[D 21:48:23.090 NotebookApp] 200 GET /kernelspecs/python2/logo-64x64.png () 1.45ms
[D 21:48:23.165 NotebookApp] 200 GET /static/components/MathJax/extensions/Safe.js?rev=2.5.3 () 0.68ms
[D 21:48:23.360 NotebookApp] Initializing websocket connection /api/kernels/5bffc82f-1765-4fc4-b3cc-3a634ac571d9/channels
[D 21:48:23.363 NotebookApp] Opening websocket /api/kernels/5bffc82f-1765-4fc4-b3cc-3a634ac571d9/channels
[D 21:48:23.363 NotebookApp] Connecting to: tcp://127.0.0.1:47444
[D 21:48:23.364 NotebookApp] Connecting to: tcp://127.0.0.1:36129
[D 21:48:23.364 NotebookApp] Connecting to: tcp://127.0.0.1:53437
```

The only way I got the websockets to connect to the kernel properly was to bypass nginx and set

``` python
c.NotebookApp.ip = '*'
```

Then, when I access it from the `https://ec2-url-for-my-instance:8888` (after clicking through the security warnings that pop up with my self-signed certificate), the websockets will connect to the kernel and I can go on my merry way running whatever I want...

I have been trying to determine why the nginx proxy isn't working for almost 2 days now.

Does anyone have any ideas about what's going on?

Contributor guide

Open the contributing guide

Research direction

Start with the /etc/nginx/conf.d/configuration.conf proxy settings and ~/.jupyter/jupyter_notebook_config.py, then compare the direct :8888 access with the failing wss /api/kernels/.../channels request. Use the browser console and NotebookApp DEBUG log entries around the 504 handshake to trace the failure. Done means the proxied kernel WebSocket connects and notebook cells execute.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, nginx, python
Domain
backend, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.