dask / dask/distributed

Priority between Clients

Open
#4,610 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I would want to be able to have some fairness between clients.

Use case:
- A cluster is running indefinitely
- Client A connects and submits a job which will take 2 hours to execute
- 10 minutes later Client B connects and submits a job which would take 1 minute to finish

Client B now must wait for 2 hours before it will get access to a worker to execute its task.

My suggestion is that the worker selects tasks in a round robin fashion from different clients to achieve a very simple fairness between clients, so small jobs can jump ahead in the queue.

In practice in our environment we have jobs that take between a few minutes and several days to complete.

I have not made any modifications to the DASK source code, but is happy to have a look if this feature would be valuable to someone else. Can you give some pointers on where it makes sense to implement this, and which approach you would recommend. Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.