dask / dask/distributed

Change the memory-limit on the fly

Open
#5,367 4 comments 0 reactions 0 assignees View on GitHub
enhancement memory
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

# Use case

This has been raised offline by a power user. Their containerization / virtualization system allows changing the amount of RAM mounted on the host on the fly. They would like to do so and then to change the memory-limit of the Worker without restarting it. Everything chained to it (target, spill, pause, and terminate thresholds) should also be recalculated.

# Current state of the art
- It is incidentally possible to update spill and pause thresholds on the fly on the worker.
- Updating the target threshold does nothing.
- Nannies are not easily accessible from the client so the terminate threshold can't be changed.
- Updating the memory_limit incidentally works for the purpose of recalculating the absolute spill and pause, but not for target or terminate.

# Proposed implementation
Calling
```python
def set_memory_limit(dask_worker, n):
dask_worker.memory_limit = n
client.run(set_memory_limit, n, workers=[...])
```
_should_ just work. As this is a niche use case, a dedicated Client API is probably overkill.
Target, spill, pause and terminate thresholds must automatically be recalculated.

# Notes

- The target threshold % is multiplied by the memory_limit by ``Worker.__init__`` to get an absolute target and then used to build the zict SpillBuffer. Need to find a way to change the zict target on the fly; this likely will require an upstream patch or at the very least additional upstream unit tests.
- The terminate threshold is stored on the Nanny, so some sort of Worker->Nanny RPC will be necessary.
- A *reduction* in the memory limit may send the worker immediately above the terminate threshold. There must be some sort of algorithm that lets it sit in paused state for a while instead. e.g. disable terminate entirely for X seconds (configurable) after a reduction.

# AC
- A straightforward way to change the memory_limit is clearly documented and covered by unit tests
- Target, spill, pause and terminate thresholds are recalculated automatically. This is covered by unit tests.
- Explicit management for the use case of reduction causing the worker to suddenly exceed the terminate threshold is implemented, documented, and covered by unit tests. As this is a new feature, it is reasonable to leave this last point as a separate, later PR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.