dask / dask/distributed

Submitting tasks from tasks

Open
#1,424 1 comment 3 reactions 0 assignees View on GitHub
discussion enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I have seen the effort in #432, and the possibility for tasks to use `get_client()` to use dask themselves. However this is awkward as it requires the task-submitting-task to stay alive just to wait for results and pass them downstream.

Wouldn't it make sense for a task to be able to return `Future` or `Delayed` object to signify to dask that the result of those computations is the result of the task? Currently the Delayed object is simply passed through to the original caller.

Example weirdness (and this is in the simple case where each task only creates a single task):

```python
def fibo(n):
if n <= 1:
return 1
delay = delayed(fibo)(n - 1) + delayed(fibo)(n - 2)
return delay

result = client.compute(delayed(fibo)(5)).result()
while isinstance(result, Delayed): # Got to keep executing until we get the actual result
result = client.compute(result).result()
result
```

-----

Apologies if this has been discussed in the past, I couldn't find this exact proposal anywhere. It is a pretty common pattern in asynchronous programming for deferred computations to return futures themselves, so I was surprise not to find it in dask.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.