dask / dask/distributed

Workloads where graph topology depends on computed values/ dynamic graph topology

Open
#3,514 1 comment 0 reactions 0 assignees View on GitHub
discussion enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

This is a design question, potentially asking the same question as https://github.com/dask/distributed/issues/1663

For some use cases (finance for my example), the work of figuring out the computing graph topology can be complex/expensive. Usually this translates into the final graph topology being dependent on some node(s) being computed , which today can be handled either all on the client side, or in dask using the secede/rejoin api
https://docs.dask.org/en/latest/futures.html#submit-tasks-from-tasks

However using this api feels quite unnatural, and using it requires all "seceded"/"paused" work to remain in memory (and occupy a thread). Same (and worse for memory usage) if all done on the client, with the added issue of data moving back and forth between client/cluster.

Have other people felt the need for a better way to handle dynamic topologies?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.