dask / dask/distributed

Restrict task/actor placement based on free worker memory

Open
#2,843 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

First off, thank you all for this great framework!

I ran into the following issue:
I want to distribute actors which consume a significant amount of RAM. For instance, each worker holds a large NumPy array ~ 1GB in memory, thus considering available worker memory is crucial for actor placement.
Furthermore, let's assume the following workers:

- worker 1: 1.5 GB memory limit
- worker 2: 3.5 GB memory limit

Hence, worker 1 can hold a maximum of 1 actor; worker 2 can hold a maximum of 3 actors (ignoring any spill, pause, terminate for now).

Unfortunately, `resources={...}` are only considered while processing a task or actor. Therefore, submitting 4 actors results in the following **invalid** placement:

- worker 1: 2 actors
- worker 2: 2 actors

As a result, worker 1 will die due to memory-limit violation. So I asked myself if there is any possibility to take available worker memory into account when assigning an actor to a worker. If there is no solution yet, I'd love to open a PR.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.