Set worker memory limit to current free memory, not total memory?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
In https://github.com/coiled/feedback/issues/185, we found that system services were taking up enough memory that the worker setting its memory limit to to the total amount of virtual memory available often caused it to put the system under memory pressure and freeze it before the terminate limit could kick in.
Perhaps dask should take this approach in general, and rather than setting its default memory limit to the total system memory (which is always going to be too high), it should use the amount of currently-available memory (maybe plus its own current RSS)?
We'd just change `.total` here to `.available`:
https://github.com/dask/distributed/blob/969aa4636bff9a4c9e1f3805fdaf9cc4d44ab1db/distributed/system.py#L19
cc @fjetter @ntabris
Contributor guide
Assessment
This issue has not been assessed yet.