dask / dask/distributed

Slow down worker when memory limit greatly exceeded

Open
#1,392 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Sometimes we underestimate memory use by resident data, or users use more memory than we anticipated in their functions. We can identify these cases with periodic checks with `psutil`, but we don't currently act.

One solution was in https://github.com/dask/distributed/pull/1235 , which started pushing more data to disk.

We might also consider just shutting down the ThreadPoolExecutor during this period until more memory is available.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.