Workloads where graph topology depends on computed values/ dynamic graph topology
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
This is a design question, potentially asking the same question as https://github.com/dask/distributed/issues/1663
For some use cases (finance for my example), the work of figuring out the computing graph topology can be complex/expensive. Usually this translates into the final graph topology being dependent on some node(s) being computed , which today can be handled either all on the client side, or in dask using the secede/rejoin api
https://docs.dask.org/en/latest/futures.html#submit-tasks-from-tasks
However using this api feels quite unnatural, and using it requires all "seceded"/"paused" work to remain in memory (and occupy a thread). Same (and worse for memory usage) if all done on the client, with the added issue of data moving back and forth between client/cluster.
Have other people felt the need for a better way to handle dynamic topologies?
Contributor guide
Assessment
This issue has not been assessed yet.