dask / dask/distributed

Worker status page: IndexError: deque index out of range

Open
#8,435 1 comment 0 reactions 0 assignees View on GitHub
bug needs info
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:

Accessing the dask worker /status page, eg:

![CleanShot 2024-01-01 at 10 14 39@2x](https://github.com/dask/distributed/assets/125105/01e7bd8a-0991-4cd4-80e1-248d3c645991)

Causes the logs to be flooded with the following multiple times a second
```
2023-12-31 23:08:32,641 - bokeh.util.tornado - ERROR - Error thrown from periodic callback:
2023-12-31 23:08:32,641 - bokeh.util.tornado - ERROR - Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 530, in callback
result_list.append(f.result())
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/bokeh/server/session.py", line 95, in _needs_document_lock_wrapper
result = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/bokeh/server/session.py", line 229, in with_document_locked
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 450, in wrapper
return invoke_with_curdoc(doc, invoke)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 408, in invoke_with_curdoc
return f()
^^^
File "/usr/local/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 449, in invoke
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/dashboard/components/__init__.py", line 41, in
doc.add_periodic_callback(lambda: update(ref), interval)
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/bokeh/core/property/validation.py", line 95, in func
return input_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/dashboard/components/__init__.py", line 49, in update
comp.update()
File "/usr/local/lib/python3.11/site-packages/bokeh/core/property/validation.py", line 95, in func
return input_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 758, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/dashboard/components/worker.py", line 189, in update
transfer_incoming_log = [
^
File "/usr/local/lib/python3.11/site-packages/distributed/dashboard/components/worker.py", line 190, in
transfer_incoming_log[-i].copy() for i in range(1, n + 1)
~~~~~~~~~~~~~~~~~~~~~^^^^
IndexError: deque index out of range
```

**Minimal Complete Verifiable Example**:

```python
# Put your MCVE code here
```

**Anything else we need to know?**:

dask is running in kubernetes and I'm port forwarding the worker's http port to view the status page

**Environment**:

- Dask version: see below
- Python version: 3.11.
- Operating System: linux
- Install method (conda, pip, source): pip

bokeh==2.4.3
dask==2023.3.2
dask-kubernetes==2023.3.2
distributed==2023.3.2.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.