dask / dask/distributed

Maintain task graph on Client

Open
#594 2 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently when the user submits jobs to the scheduler it sends up little sub-graphs, stores Futures for the results, and forgets the original computation. Instead, we could maintain an active `.dask` attribute on the Client. This would provide the following benefits:
1. It would be useful if the scheduler goes down and comes back up. We can resubmit all of the computations that we care about.
2. It would allow for local debugging. If a task were to fail we could bring the necessary data to the client and rerun the affected task, allowing for local `pdb` style debugging.
3. It would be useful for general understanding among expert users.

However, this isn't free. We would need to add some code to all of the task submission functions (`submit`, `map`, `compute`, `persist`) though this can probably be done just with the `Client._graph_to_futures` method. We would also need to add code on garbage collection.

My main motivation for this is to help address the High Availability question (benefit 1 above).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.