dask / dask/distributed

Dynamically adjusting priority via Futures

Open
#1,753 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Basically copied from [this comment]( https://github.com/dask/distributed/pull/1651#issuecomment-359084241 ) in PR ( https://github.com/dask/distributed/pull/1651 ). Also related to issue ( https://github.com/dask/dask/issues/2860 ).

One thing that would be nice is to be able to propagate priority changes from `Future`s themselves. Namely if one tries to call `result`, it would be good to bump the priority. Maybe this could be configurable if not generally desirable.

To provide an example use case, imagine one has a Dask object (Array, Dataframe, etc.) and has called `persist` on the object. Then one wants to inspect a small piece of it while the computation proceeds. So they select out that piece and call `compute`. It would be good if this already was interpreted as the user wanting this piece of the result sooner than the rest. That way the user can play with this piece as the rest of the computation completes. If needed, this process can be repeated with other pieces.

cc @mrocklin @p-himik

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.