Submitting tasks from tasks
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I have seen the effort in #432, and the possibility for tasks to use `get_client()` to use dask themselves. However this is awkward as it requires the task-submitting-task to stay alive just to wait for results and pass them downstream.
Wouldn't it make sense for a task to be able to return `Future` or `Delayed` object to signify to dask that the result of those computations is the result of the task? Currently the Delayed object is simply passed through to the original caller.
Example weirdness (and this is in the simple case where each task only creates a single task):
```python
def fibo(n):
if n <= 1:
return 1
delay = delayed(fibo)(n - 1) + delayed(fibo)(n - 2)
return delay
result = client.compute(delayed(fibo)(5)).result()
while isinstance(result, Delayed): # Got to keep executing until we get the actual result
result = client.compute(result).result()
result
```
-----
Apologies if this has been discussed in the past, I couldn't find this exact proposal anywhere. It is a pretty common pattern in asynchronous programming for deferred computations to return futures themselves, so I was surprise not to find it in dask.
Contributor guide
Assessment
This issue has not been assessed yet.