dask / dask/distributed

Making AMM `ReduceReplicas` less aggressive towards widely-shared dependencies

Open
#6,056 2 comments 0 reactions 0 assignees View on GitHub
discussion enhancement memory performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Corollary to https://github.com/dask/distributed/issues/6038. In that issue, I described a situation where workers thought a key (which most tasks depended on) had 82 replicas, but in reality it only had 1.

This issue is about the fact that `ReduceReplicas` maybe shouldn't try to delete copies of that critical key so aggressively.
```
* * * * * *
\ \ \ / / /
x y
```
In this case `x` and `y` are going to be reused by every task, so they will end up having replicas on most workers. Constantly deleting them is inefficient—as soon as you delete it, the next task that wants to run on that worker is going to have to transfer it back again.

(Of course, once most of the `*` tasks are done, then you should start reducing replicas. But while the cluster is fully saturated with `*` tasks, there's no benefit to doing this.)

I'm not sure what metric to use for this. Ideas explored in https://github.com/dask/distributed/pull/4967, https://github.com/dask/distributed/pull/5325, https://github.com/dask/distributed/pull/5326 could be interesting here.

Really, this issue is just about how to calculate a smarter target for this `desired_replicas` count automatically based on the task's `waiters`, number of current workers, etc.: https://github.com/dask/distributed/blob/4b3e0c2595d29dc0069fbfb6d2cff8bd3fa83949/distributed/active_memory_manager.py#L477

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.