Change the memory-limit on the fly
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
# Use case
This has been raised offline by a power user. Their containerization / virtualization system allows changing the amount of RAM mounted on the host on the fly. They would like to do so and then to change the memory-limit of the Worker without restarting it. Everything chained to it (target, spill, pause, and terminate thresholds) should also be recalculated.
# Current state of the art
- It is incidentally possible to update spill and pause thresholds on the fly on the worker.
- Updating the target threshold does nothing.
- Nannies are not easily accessible from the client so the terminate threshold can't be changed.
- Updating the memory_limit incidentally works for the purpose of recalculating the absolute spill and pause, but not for target or terminate.
# Proposed implementation
Calling
```python
def set_memory_limit(dask_worker, n):
dask_worker.memory_limit = n
client.run(set_memory_limit, n, workers=[...])
```
_should_ just work. As this is a niche use case, a dedicated Client API is probably overkill.
Target, spill, pause and terminate thresholds must automatically be recalculated.
# Notes
- The target threshold % is multiplied by the memory_limit by ``Worker.__init__`` to get an absolute target and then used to build the zict SpillBuffer. Need to find a way to change the zict target on the fly; this likely will require an upstream patch or at the very least additional upstream unit tests.
- The terminate threshold is stored on the Nanny, so some sort of Worker->Nanny RPC will be necessary.
- A *reduction* in the memory limit may send the worker immediately above the terminate threshold. There must be some sort of algorithm that lets it sit in paused state for a while instead. e.g. disable terminate entirely for X seconds (configurable) after a reduction.
# AC
- A straightforward way to change the memory_limit is clearly documented and covered by unit tests
- Target, spill, pause and terminate thresholds are recalculated automatically. This is covered by unit tests.
- Explicit management for the use case of reduction causing the worker to suddenly exceed the terminate threshold is implemented, documented, and covered by unit tests. As this is a new feature, it is reasonable to leave this last point as a separate, later PR.
Contributor guide
Assessment
This issue has not been assessed yet.