Set scheduler log sizes automatically based on available memory
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
There are frequent reports of scheduler memory growing over time:
* https://github.com/dask/distributed/issues/5509
* https://github.com/dask/distributed/issues/4987#issuecomment-877473670
* https://github.com/dask/distributed/issues/3898 (there is a different problem here; memory is leaking even with logs turned off, but turning off logs was necessary to debug)
* https://github.com/dask/distributed/issues/4998
They often involve memory graphs that look like:


It's very likely that there is a real bug in the scheduler causing memory to accumulate (https://github.com/dask/distributed/issues/3898#issuecomment-814444832), but often the steep slope on these graphs is caused by various logs on the scheduler accumulating, such as:
* `transition_log` - `distributed.scheduler.transition-log-length`
* `log` - `distributed.scheduler.transition-log-length` (should maybe be `distributed.admin.log-length`?)
* `events` - `distributed.scheduler.events-log-length`
* `computations` - `distributed.diagnostics.computations.max-history`
* `Node._deque_handler` - `distributed.admin.log-length`
I propose two things:
1. Log lengths should be set as a percentage of available memory, not as a length—this is much easier for users to configure
Note that for some/most of these, that may be difficult to do accurately, since the size of the entries is unknown. A rough estimate is probably okay.
2. A memory-cleanup callback that runs, say, once a second, and clears our excess logs if the scheduler is under memory pressure.
Contributor guide
Assessment
This issue has not been assessed yet.