dask / dask/distributed

Dashboard memory/CPU increases when dashboard is used, making scheduler unresponsive

Open
#6,103 11 comments 0 reactions 0 assignees View on GitHub
bug diagnostics good second issue p2 performance regression
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:

Running Dask-Scheduler on a dedicated host, with dashboard enabled and used, results in CPU utilization increase to the point where scheduler slows down to a halt.

**What you expected to happen**:

Using dashboard should not result in so much CPU utilization, resulting in unresponsive scheduler.

If dashboard is not intended to be used in a production environment, a warning must be added to documentation (or even the CLI) to inform the users.

**Minimal Complete Verifiable Example**:

1. Use any recent release (at least for the past year, including main dev branch).
2. Install in a venv to make sure all requirements are installed with the most recent available version
3. Start `dask-scheduler` with no extra CLI parameters
4. Don't even need to start a worker, just scheduler by itself is enough to reproduce the problem
5. Visit the dashboard, browse to multiple pages
6. Observe that with each page click, memory and CPU utilization increase

**Anything else we need to know?**:

In a production environment, we have to constantly monitor the scheduler machine and restart `dask-scheduler` due to this issue.
Not sure if it helps to know: when CPU usage goes above 50%, scheduler stops doing some optimizations: https://github.com/dask/distributed/blob/main/distributed/scheduler.py#L8048

**Environment**:

- Dask version: Many, including latest `2022.4.0`
- Python version: Tested with 3.8.8 and 3.9.7
- Operating System: Tested with Ubuntu 18.04 and MacOS 12.3.1
- Install method (conda, pip, source): tested with both pip and source

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.