dask / dask/distributed

`Scheduler.get_worker_monitor_info` hangs until workers are killed

Open
#5,317 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
Async calls to `Scheduler.get_worker_monitor_info` sometimes hang and must be interrupted with Ctrl+C, causing all workers to restart.

**What you expected to happen**:
I would expect to be able to make calls to `get_worker_monitor_info` without hanging.

**Minimal Complete Verifiable Example**:
This happens consistently when trying to make the async calls after performing some tasks:

```python
import dask
from distributed import Client

client = Client()

# do a basic computation
df = dask.datasets.timeseries()
df.sort_values(by="id").compute()

# try to get worker metrics
metrics = await client.cluster.scheduler.get_worker_monitor_info()
```

I'm also able to get this behavior by making repeated calls to `get_worker_monitor_info`:

```python
In [1]: from distributed import Client
...:
...: client = Client()
...:
...: metrics = await client.cluster.scheduler.get_worker_monitor_info() # finishes successfully

In [2]: metrics = await client.cluster.scheduler.get_worker_monitor_info() # hangs
```

**Environment**:

- Dask version: latest `main`
- Python version: 3.8
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): source

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.