Dynamically adjusting priority via Futures
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Basically copied from [this comment]( https://github.com/dask/distributed/pull/1651#issuecomment-359084241 ) in PR ( https://github.com/dask/distributed/pull/1651 ). Also related to issue ( https://github.com/dask/dask/issues/2860 ).
One thing that would be nice is to be able to propagate priority changes from `Future`s themselves. Namely if one tries to call `result`, it would be good to bump the priority. Maybe this could be configurable if not generally desirable.
To provide an example use case, imagine one has a Dask object (Array, Dataframe, etc.) and has called `persist` on the object. Then one wants to inspect a small piece of it while the computation proceeds. So they select out that piece and call `compute`. It would be good if this already was interpreted as the user wanting this piece of the result sooner than the rest. That way the user can play with this piece as the rest of the computation completes. If needed, this process can be repeated with other pieces.
cc @mrocklin @p-himik
Contributor guide
Assessment
This issue has not been assessed yet.