dask / dask/distributed

AMM: select candidates based on memory percentage, not absolute usage?

Open
#5,733 3 comments 0 reactions 0 assignees View on GitHub
memory
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The `ActiveMemoryManagerExtension` currently selects candidates to receive or drop replicates based on the workers with the lowest/highest optimistic memory, in bytes.

In heterogeneous clusters, doesn't this bias AMM towards small workers being under more memory pressure than large workers?

For example:
* worker A is using 5GiB, and has 10GiB capacity
* worker B is using 8GiB, and has 100GiB capacity

`find_recipient` would pick worker A to replicate keys onto, even though it's under more memory pressure.
`find_dropper` would pick worker B to drop from, even though it's under much less memory pressure.

cc @crusaderky

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.