ipython / ipython/ipykernel

Kernel crashes randomly: `IndexError: pop from an empty deque`

未关闭
#1,250 2 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
734
派生
412
平均合并
1 天 5 小时
30 天内合并 PR
8

描述

## Description

Hello,
We encounter a random error which occurs from time to time on our shared jupyterhub instance.
After some time of running many jupyterlab notebooks, the kernel tends the crash always at the same cell of a notebook(which changes from a crash occurrence to another). Note that the same cell works in other runs, and that the error does not occur on the same cell or cell content after the jupyterhub is restarted and the error happens again.
Here is the stacktrace we get:
```
Traceback (most recent call last):
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/ipykernel_launcher.py", line 18, in
app.launch_new_instance()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 205, in start
self.asyncio_loop.run_forever()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
self._run_once()
File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
handle = self._ready.popleft()
IndexError: pop from an empty deque
```

To overcome the error, we need to restart the jupyterhub instance thus interrupting everybody's work.

## Reproduce

1. Open a notebook
2. Run a series of cells
3. A popup appears telling that the kernel has crashed and need to be restarted

## Expected behavior

The kernel does not crash and all cells are run successfully.

## Context

- Operating System and version: Debian 11 (bullseye)
- Browser and version: Chrome 125, firefox 125
- JupyterLab version: 4.1.2

For reference, here are all jupyter-related packages that are used to build our jupyterhub instance
```
jupyter-client==8.6.0 \
--hash=sha256:0642244bb83b4764ae60d07e010e15f0e2d275ec4e918a8f7b80fbbef3ca60c7 \
--hash=sha256:909c474dbe62582ae62b758bca86d6518c85234bdee2d908c778db6d72f39d99
# via
# ipykernel
# jupyter-server
# nbclient
jupyter-core==5.7.1 \
--hash=sha256:c65c82126453a723a2804aa52409930434598fd9d35091d63dfb919d2b765bb7 \
--hash=sha256:de61a9d7fc71240f688b2fb5ab659fbb56979458dc66a71decd098e03c79e218
# via
# ipykernel
# jupyter-client
# jupyter-server
# jupyterlab
# nbclient
# nbconvert
# nbformat
jupyter-events==0.9.0 \
--hash=sha256:81ad2e4bc710881ec274d31c6c50669d71bbaa5dd9d01e600b56faa85700d399 \
--hash=sha256:d853b3c10273ff9bc8bb8b30076d65e2c9685579db736873de6c2232dde148bf
# via jupyter-server
jupyter-lsp==2.2.2 \
--hash=sha256:256d24620542ae4bba04a50fc1f6ffe208093a07d8e697fea0a8d1b8ca1b7e5b \
--hash=sha256:3b95229e4168355a8c91928057c1621ac3510ba98b2a925e82ebd77f078b1aa5
# via
# jupyterlab
# jupyterlab-lsp
jupyter-resource-usage==1.0.1 \
--hash=sha256:d722ad32fc8bfaff3f81da4f8a2202c5e5258895d546399ff0e5ddf11f56bd8e \
--hash=sha256:ede723ebb63d531615d0da4f8769470c8ee9cc3fa18e6af9a12b70f11972cc61
# via -r requirements.txt
jupyter-server==2.12.5 \
--hash=sha256:0edb626c94baa22809be1323f9770cf1c00a952b17097592e40d03e6a3951689 \
--hash=sha256:184a0f82809a8522777cfb6b760ab6f4b1bb398664c5860a27cec696cb884923
# via
# jupyter-lsp
# jupyter-resource-usage
# jupyterlab
# jupyterlab-server
# notebook
# notebook-shim
jupyter-server-terminals==0.5.2 \
--hash=sha256:1b80c12765da979513c42c90215481bbc39bd8ae7c0350b4f85bc3eb58d0fa80 \
--hash=sha256:396b5ccc0881e550bf0ee7012c6ef1b53edbde69e67cab1d56e89711b46052e8
# via jupyter-server
jupyter-telemetry==0.1.0 \
--hash=sha256:1de3e423b23aa40ca4a4238d65c56dda544061ff5aedc3f7647220ed7e3b9589 \
--hash=sha256:445c613ae3df70d255fe3de202f936bba8b77b4055c43207edf22468ac875314
# via jupyterhub
jupyterhub==4.0.2 \
--hash=sha256:2f389e7d3067e1b11bb4091719048eedecee161039fd2e5b025d031f7ab23c62 \
--hash=sha256:d4e450eed8d90dfbcf0eca08f00f2093a0bce74dc51f7cfc0b7057f602341a1d
# via
# -r requirements.txt
# jupyterhub-traefik-proxy
# sudospawner
jupyterhub-traefik-proxy==1.1.0 \
--hash=sha256:6972d8eb788274d1d541c11cc10764d94a6829843bdb15873b60e19519bd4d5c \
--hash=sha256:9e67eb354165b9400fb5a01c4272b4c2073dbd81fbbcd4e766ea7a62fac12cde
# via -r requirements.txt
jupyterlab==4.1.2 \
--hash=sha256:5d6348b3ed4085181499f621b7dfb6eb0b1f57f3586857aadfc8e3bf4c4885f9 \
--hash=sha256:aa88193f03cf4d3555f6712f04d74112b5eb85edd7d222c588c7603a26d33c5b
# via
# -r requirements.txt
# jupyterlab-lsp
# notebook
jupyterlab-lsp==5.0.3 \
--hash=sha256:1a1c96f60202e49c28538f40b3ee19487d490a338bed88c95368073412689e22 \
--hash=sha256:a9c8a3a646494be484d7e3174ab1d2415ed60f4773cec8479aa5ba37e5f04bc9
# via -r requirements.txt
jupyterlab-pygments==0.3.0 \
--hash=sha256:721aca4d9029252b11cfa9d185e5b5af4d54772bb8072f9b7036f4170054d35d \
--hash=sha256:841a89020971da1d8693f1a99997aefc5dc424bb1b251fd6322462a1b8842780
# via nbconvert
jupyterlab-server==2.25.3 \
--hash=sha256:846f125a8a19656611df5b03e5912c8393cea6900859baa64fa515eb64a8dc40 \
--hash=sha256:c48862519fded9b418c71645d85a49b2f0ec50d032ba8316738e9276046088c1
# via
# jupyterlab
# notebook
jupyterlab-skip-traceback==5.0.0 \
--hash=sha256:a1091b639850ef3b614b534816f38647c41a2519b4bf47ca7e7585770fb3dd9e \
--hash=sha256:dd1acf9150c951150c6de24ebc6c3247bb7788971671158cc0915fd3310ed75b
# via -r requirements.txt
jupyterlab-widgets==3.0.10 \
```

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.

You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

贡献指南

打开贡献指南

调研方向

Start with the traceback in Python's asyncio/base_events.py, then inspect the ipykernel entry points in ipykernel_launcher.py and kernelapp.py. Try to reproduce the intermittent crash by running notebook cells repeatedly in the listed JupyterLab 4.1.2 environment. Done means identifying why the kernel reaches an empty ready deque and preventing the kernel crash without requiring a JupyterHub restart.

由索引模型根据 Issue 内容生成。

评估

技术栈
jupyter-notebook, python
领域
backend
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。