dask / dask/distributed

Unresponsive workers should be flagged on the dashboard

Open
#8,546 1 comment 0 reactions 0 assignees View on GitHub
dashboard
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

- Related to #8537

Let's hamstring a worker:
```python
async def kill_event_loop():
while True:
pass

fut = c.submit(kill_event_loop)
```
This makes the worker completely unresponsive. After 5 minutes (`distributed.scheduler.worker-ttl`) without a single heartbeat coming through, the scheduler will disconnect it forcefully.

Until that happens, this is what I'm seeing on the dashboard:
![Screenshot from 2024-02-29 17-26-34](https://github.com/dask/distributed/assets/6213168/123a47d8-777e-4b66-b41f-bb0439f5d5b0)

# Desired behaviour
There should be an indication on the dashboard of how many seconds have passed since the last heartbeat.
There should be also be visual cues - e.g. the whole line turning red - to indicate when you get to a significant threshold since the last heartbeat - e.g. max(4x of the expected heartbeat ratio, 25% of the worker-ttl)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.