ipython / ipython/ipykernel

Kernel crashes randomly: `IndexError: pop from an empty deque`

Aperta
#1,250 2 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
734
Fork
412
Merge medio
1g 5h
PR unite (30g)
8

Descrizione

## Description

Hello,
We encounter a random error which occurs from time to time on our shared jupyterhub instance.
After some time of running many jupyterlab notebooks, the kernel tends the crash always at the same cell of a notebook(which changes from a crash occurrence to another). Note that the same cell works in other runs, and that the error does not occur on the same cell or cell content after the jupyterhub is restarted and the error happens again.
Here is the stacktrace we get:
```
Traceback (most recent call last):
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/ipykernel_launcher.py", line 18, in
app.launch_new_instance()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 205, in start
self.asyncio_loop.run_forever()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
self._run_once()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
handle = self._ready.popleft()
IndexError: pop from an empty deque
```

To overcome the error, we need to restart the jupyterhub instance thus interrupting everybody's work.

## Reproduce

1. Open a notebook
2. Run a series of cells
3. A popup appears telling that the kernel has crashed and need to be restarted

## Expected behavior

The kernel does not crash and all cells are run successfully.

## Context

- Operating System and version: Debian 11 (bullseye)
- Browser and version: Chrome 125, firefox 125
- JupyterLab version: 4.1.2

For reference, here are all jupyter-related packages that are used to build our jupyterhub instance
```
jupyter-client==8.6.0 \
--hash=sha256:0642244bb83b4764ae60d07e010e15f0e2d275ec4e918a8f7b80fbbef3ca60c7 \
--hash=sha256:909c474dbe62582ae62b758bca86d6518c85234bdee2d908c778db6d72f39d99
# via
# ipykernel
# jupyter-server
# nbclient
jupyter-core==5.7.1 \
--hash=sha256:c65c82126453a723a2804aa52409930434598fd9d35091d63dfb919d2b765bb7 \
--hash=sha256:de61a9d7fc71240f688b2fb5ab659fbb56979458dc66a71decd098e03c79e218
# via
# ipykernel
# jupyter-client
# jupyter-server
# jupyterlab
# nbclient
# nbconvert
# nbformat
jupyter-events==0.9.0 \
--hash=sha256:81ad2e4bc710881ec274d31c6c50669d71bbaa5dd9d01e600b56faa85700d399 \
--hash=sha256:d853b3c10273ff9bc8bb8b30076d65e2c9685579db736873de6c2232dde148bf
# via jupyter-server
jupyter-lsp==2.2.2 \
--hash=sha256:256d24620542ae4bba04a50fc1f6ffe208093a07d8e697fea0a8d1b8ca1b7e5b \
--hash=sha256:3b95229e4168355a8c91928057c1621ac3510ba98b2a925e82ebd77f078b1aa5
# via
# jupyterlab
# jupyterlab-lsp
jupyter-resource-usage==1.0.1 \
--hash=sha256:d722ad32fc8bfaff3f81da4f8a2202c5e5258895d546399ff0e5ddf11f56bd8e \
--hash=sha256:ede723ebb63d531615d0da4f8769470c8ee9cc3fa18e6af9a12b70f11972cc61
# via -r requirements.txt
jupyter-server==2.12.5 \
--hash=sha256:0edb626c94baa22809be1323f9770cf1c00a952b17097592e40d03e6a3951689 \
--hash=sha256:184a0f82809a8522777cfb6b760ab6f4b1bb398664c5860a27cec696cb884923
# via
# jupyter-lsp
# jupyter-resource-usage
# jupyterlab
# jupyterlab-server
# notebook
# notebook-shim
jupyter-server-terminals==0.5.2 \
--hash=sha256:1b80c12765da979513c42c90215481bbc39bd8ae7c0350b4f85bc3eb58d0fa80 \
--hash=sha256:396b5ccc0881e550bf0ee7012c6ef1b53edbde69e67cab1d56e89711b46052e8
# via jupyter-server
jupyter-telemetry==0.1.0 \
--hash=sha256:1de3e423b23aa40ca4a4238d65c56dda544061ff5aedc3f7647220ed7e3b9589 \
--hash=sha256:445c613ae3df70d255fe3de202f936bba8b77b4055c43207edf22468ac875314
# via jupyterhub
jupyterhub==4.0.2 \
--hash=sha256:2f389e7d3067e1b11bb4091719048eedecee161039fd2e5b025d031f7ab23c62 \
--hash=sha256:d4e450eed8d90dfbcf0eca08f00f2093a0bce74dc51f7cfc0b7057f602341a1d
# via
# -r requirements.txt
# jupyterhub-traefik-proxy
# sudospawner
jupyterhub-traefik-proxy==1.1.0 \
--hash=sha256:6972d8eb788274d1d541c11cc10764d94a6829843bdb15873b60e19519bd4d5c \
--hash=sha256:9e67eb354165b9400fb5a01c4272b4c2073dbd81fbbcd4e766ea7a62fac12cde
# via -r requirements.txt
jupyterlab==4.1.2 \
--hash=sha256:5d6348b3ed4085181499f621b7dfb6eb0b1f57f3586857aadfc8e3bf4c4885f9 \
--hash=sha256:aa88193f03cf4d3555f6712f04d74112b5eb85edd7d222c588c7603a26d33c5b
# via
# -r requirements.txt
# jupyterlab-lsp
# notebook
jupyterlab-lsp==5.0.3 \
--hash=sha256:1a1c96f60202e49c28538f40b3ee19487d490a338bed88c95368073412689e22 \
--hash=sha256:a9c8a3a646494be484d7e3174ab1d2415ed60f4773cec8479aa5ba37e5f04bc9
# via -r requirements.txt
jupyterlab-pygments==0.3.0 \
--hash=sha256:721aca4d9029252b11cfa9d185e5b5af4d54772bb8072f9b7036f4170054d35d \
--hash=sha256:841a89020971da1d8693f1a99997aefc5dc424bb1b251fd6322462a1b8842780
# via nbconvert
jupyterlab-server==2.25.3 \
--hash=sha256:846f125a8a19656611df5b03e5912c8393cea6900859baa64fa515eb64a8dc40 \
--hash=sha256:c48862519fded9b418c71645d85a49b2f0ec50d032ba8316738e9276046088c1
# via
# jupyterlab
# notebook
jupyterlab-skip-traceback==5.0.0 \
--hash=sha256:a1091b639850ef3b614b534816f38647c41a2519b4bf47ca7e7585770fb3dd9e \
--hash=sha256:dd1acf9150c951150c6de24ebc6c3247bb7788971671158cc0915fd3310ed75b
# via -r requirements.txt
jupyterlab-widgets==3.0.10 \
```

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.

You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the traceback in Python's asyncio/base_events.py, then inspect the ipykernel entry points in ipykernel_launcher.py and kernelapp.py. Try to reproduce the intermittent crash by running notebook cells repeatedly in the listed JupyterLab 4.1.2 environment. Done means identifying why the kernel reaches an empty ready deque and preventing the kernel crash without requiring a JupyterHub restart.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
jupyter-notebook, python
Ambito
backend
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.