dask / dask/distributed

Reduce activity when CPU is under load

Open
#4,053 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Oversubscription of threads is a common problem, especially when tasks call routines that are themselves multi-threaded. The most common culprit of this is BLAS/LAPACK calls.

A couple of years ago I think that the Joblib folks did something clever, they polled the process to see if it was oversubscribed, and if it was, it slowed down work. We already check CPU metrics regularly with `psutil`. The `Worker.ensure_computing` method could choose to hold off on submitting new tasks if it notices that

1. The local process' CPU load is already above where it should be.
2. There is at least one task already running (we need to ensure that we don't get into a situation where we block all forward progress)

This has come up several times, but most recently came up in a post here: https://coiled.io/blog/bomb-detection-with-dask-and-machine-learning/

https://github.com/dask/distributed/blob/67a9a5963b757835d185c7b202f6895069934f97/distributed/worker.py#L2465-L2499

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.