dask / dask/distributed

Clarification of the task graph: are futures allowed in tuples that aren't tasks?

Open
#4,183 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

As reported by https://github.com/dask/distributed/issues/4177, [custreamz CI](https://gpuci.gpuopenanalytics.com/view/RAPIDS%20-%20GPU%20Matrix/job/rapidsai/job/gpuci/job/cudf/job/branches/job/cudf-gpu-matrix-branch-0.16/CUDA=10.1,LINUX_VERSION=ubuntu16.04,PYTHON_VERSION=3.7/49/testReport/custreamz.tests/test_dataframes/test_getitem_dask_/) fails because the task graph contains tuples with future objects.

The question is if this is even allowed? Dask don't allow keys in tuples that aren't tasks. E.g. in the following `("x",)` is interpreted as a literal with no dependencies:
```python
import dask.core
dsk = {
"x": 1,
"a": ("x",),
"b": ["x"]
}
assert dask.core.get_dependencies(dsk, "a") == set()
assert dask.core.get_dependencies(dsk, "b") == {"x"}
```
But what about futures? Conceptually, futures are _remote_ graph keys so I am inclined to think that they should have the same semantic as regular keys.

What are you thoughts, should the following be allowed?
```python

from distributed import LocalCluster, Client

if __name__ == '__main__':
client = Client(LocalCluster(n_workers=1))

def getfirst(x):
return x[0]

x = client.submit(sum, [1,2])
y = client.submit(getfirst, (x,))
y.result()
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.