dask / dask/distributed

Ability to specify cost value for computations

Open
#8,654 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Some functions may require more resources than others so the ability to set the function cost is useful to avoid overloading workers.

Imagine function we have 1 worker with 4 threads and `F(x)`, `G(x)` while 4 instances of `G(x)` could be computed easily on the worker, 2 or more instances of `F(x)` will overload the worker CPU thus a possible API could be.

```python
f.submit(x=123, compute_cost=4)
```
This will allow the function to fill up the 4 threads that the worker has without submitting more tasks to it before completing this function.

Of course, the API requires considerable design choices but this issue serves as a discussion starter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.