Make Dask support `External` tasks
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
An `External` Task is a work unit that runs in an external environment than Dask.
But still, it is known by the scheduler even before it becomes available and can be used in a task graph as any (internal) Dask task.
An External Task is created in an `external` state.
It has a possible transition to the memory state when it becomes available in the distributed memory of Dask.
And it unblocks all the dependent tasks by triggering the task transition Algorithm.
External task makes it possible to integrate Dask distributed analytics in larger workflows implying other environments, typically this feature has been used in HPC workflows coupling MPI simulation to Dask analytics in a producer-consumer functionality.
The external tasks can can be used in other workflows such as digital twins workflows and IoT integration.
A way to do that is by using `scatter` operation from the client Class to send a block of data from the simulation process (with a client) to a Dask worker. Then sending semantic metadata to be able to use those data in the analytics.
The metadata includes the future returned by the `scatter` with semantic metadata from the simulation.
The issue with this solution is being able to submit tasks to Dask only when the data is available. Data comes incrementally from the simulation side (iteration by iteration). Thus one has to write incremental algorithms for analytics needing data from several timesteps.
Moreover, a lot of metadata has to go through the scheduler and usually more than 512 clients are needed which overloads the scheduler.
In addition to bypassing the disk accesses to get the data, the external tasks allow the submission of task graphs even before data becomes available in the worker's memory.
This feature is already implemented in a forked version of Dask [distributed](https://github.com/GueroudjiAmal/distributed)
A detailed demonstration of the feature is published [here](https://github.com/GueroudjiAmal/Dask-internals-presentation/tree/main/Demo%20External%20Tasks). And a demo is [here](https://github.com/dask/community/issues/307)
In this example, two clients have been used `Client01` submits the tasks, and `Client02` only sends the data with the same `key` and activates the external mode which is referred to as `deisa` mode for (Dask Enabled In Situ Analytics).
In addition to using the same key, the scatter operation needs to act like other finished tasks, that is triggering the transition algorithm, once done. This is mandatory as the external tasks are used in analytics even before they are available.
Contributor guide
Assessment
This issue has not been assessed yet.