Connection to notebook server could not be established.
- Dominant language
- Jupyter Notebook
- Stars
- 13.3k
- Forks
- 5.8k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 7
Description
Problem:
Kernel dies when training a tensorflow-gpu model in Jupyter Notebook after about 2 hours user time. I have observed this problem in over 25 independent 'run all' executions of the notebook cells. The error always shows up after the PUT operation by the notebook.
Environment:
The following is the Anaconda environment configuration that I use on my Windows 10 desktop
Anaconda 3.5.3
python 3.6.6
conda 4.5.11
jupyter 4.4.0
ipykernel 5.1.0
tensorflow-gpu 1.11.0
Message displayed Jupyter notebook in Chrome:
A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.
Execution from Anaconda prompt with:
jupyter notebook model1.ipynb --debug
Truncated Output:
[D 04:25:24.136 NotebookApp] activity on ee23f68b-9450-4f8a-8107-4c588bde6101: stream
[D 04:25:33.964 NotebookApp] 200 GET /api/contents/model.ipynb?content=0&_=1539411932463 (::1) 1.00ms
[I 04:25:33.971 NotebookApp] Saving file at /model.ipynb
[D 04:25:33.972 NotebookApp] Saving C:\Users\JC\Documents\Flink\problem1\model.ipynb
[D 04:25:34.218 NotebookApp] 200 PUT /api/contents/model.ipynb (::1) 249.33ms
Assertion failed: nbytes == sizeof (dummy) (bundled\zeromq\src\signaler.cpp:402)
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
(tensorflow_gpu_3.6) C:\Users\JC\Documents\Flink\problem1>
What appears to be unique about this error?:
I have gone through the other 'issues' with Jupyter notebook and there are several reports of similar notebook behavior which involve zeromq. However, the 'zeromq\src\signaler.cpp:402' error message appears to be new and perhaps the solution will depend entirely on zeromq. I do not know.
Query:
Has anyone else observed this error message ? If yes, does anyone know of a fix for this problem ?
Contributor guide
Research direction
Start by reproducing the failure with `jupyter notebook model1.ipynb --debug` using the listed Anaconda environment. Inspect the `/api/contents/model.ipynb` PUT immediately before `bundled\zeromq\src\signaler.cpp:402`, then determine whether the failure belongs to Jupyter Notebook or ZeroMQ. Done means documenting a confirmed cause and reproducible fix or workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, tensorflow
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100