dask / dask/distributed

Set scheduler log sizes automatically based on available memory

Open
#5,570 1 comment 0 reactions 0 assignees View on GitHub
diagnostics documentation stability
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

There are frequent reports of scheduler memory growing over time:
* https://github.com/dask/distributed/issues/5509
* https://github.com/dask/distributed/issues/4987#issuecomment-877473670
* https://github.com/dask/distributed/issues/3898 (there is a different problem here; memory is leaking even with logs turned off, but turning off logs was necessary to debug)
* https://github.com/dask/distributed/issues/4998

They often involve memory graphs that look like:
![image](https://user-images.githubusercontent.com/3309802/145085963-7e5b45ea-fb1b-4012-80b4-2878642ffa5e.png)
![](https://user-images.githubusercontent.com/31919686/141488527-cfc95869-19c7-4780-823d-0f8a791d1800.png)

It's very likely that there is a real bug in the scheduler causing memory to accumulate (https://github.com/dask/distributed/issues/3898#issuecomment-814444832), but often the steep slope on these graphs is caused by various logs on the scheduler accumulating, such as:
* `transition_log` - `distributed.scheduler.transition-log-length`
* `log` - `distributed.scheduler.transition-log-length` (should maybe be `distributed.admin.log-length`?)
* `events` - `distributed.scheduler.events-log-length`
* `computations` - `distributed.diagnostics.computations.max-history`
* `Node._deque_handler` - `distributed.admin.log-length`

I propose two things:
1. Log lengths should be set as a percentage of available memory, not as a length—this is much easier for users to configure
Note that for some/most of these, that may be difficult to do accurately, since the size of the entries is unknown. A rough estimate is probably okay.
2. A memory-cleanup callback that runs, say, once a second, and clears our excess logs if the scheduler is under memory pressure.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.