`Scheduler.get_worker_monitor_info` hangs until workers are killed
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
Async calls to `Scheduler.get_worker_monitor_info` sometimes hang and must be interrupted with Ctrl+C, causing all workers to restart.
**What you expected to happen**:
I would expect to be able to make calls to `get_worker_monitor_info` without hanging.
**Minimal Complete Verifiable Example**:
This happens consistently when trying to make the async calls after performing some tasks:
```python
import dask
from distributed import Client
client = Client()
# do a basic computation
df = dask.datasets.timeseries()
df.sort_values(by="id").compute()
# try to get worker metrics
metrics = await client.cluster.scheduler.get_worker_monitor_info()
```
I'm also able to get this behavior by making repeated calls to `get_worker_monitor_info`:
```python
In [1]: from distributed import Client
...:
...: client = Client()
...:
...: metrics = await client.cluster.scheduler.get_worker_monitor_info() # finishes successfully
In [2]: metrics = await client.cluster.scheduler.get_worker_monitor_info() # hangs
```
**Environment**:
- Dask version: latest `main`
- Python version: 3.8
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): source
Contributor guide
Assessment
This issue has not been assessed yet.