Slow down worker when memory limit greatly exceeded
Open
performance
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Sometimes we underestimate memory use by resident data, or users use more memory than we anticipated in their functions. We can identify these cases with periodic checks with `psutil`, but we don't currently act.
One solution was in https://github.com/dask/distributed/pull/1235 , which started pushing more data to disk.
We might also consider just shutting down the ThreadPoolExecutor during this period until more memory is available.
Contributor guide
Assessment
This issue has not been assessed yet.