dask / dask/dask

Add method to optimize collection and return HighLevelGraph

Open
#4,626 4 comments 0 reactions 0 assignees View on GitHub
highlevelgraph
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

It appears that when using `__dask_optimize__` it returns raw dicts. However the expectation has become `HighLevelGraph`s should be used instead. To meet that expectation, it would be nice if the `__dask_optimize__` functions also met this need. Here's an example with `Delayed` below, but can see similar things with the other collections.

Example:

```python
In [1]: from dask.delayed import Delayed, delayed

In [2]: d = delayed(sum)([2])

In [3]: Delayed.__dask_optimize__(d.__dask_graph__(), d.__dask_keys__())
Out[3]:
{'sum-aa5d6384-422a-4372-ada2-9753cf8efe31': (,
[2])}
```


Environment:

```yaml
name: dask
channels:
- conda-forge
- defaults
dependencies:
- appnope=0.1.0=py37_1000
- backcall=0.1.0=py_0
- blas=2.4=openblas
- bokeh=1.0.4=py37_1000
- bzip2=1.0.6=h1de35cc_1002
- ca-certificates=2019.3.9=hecc5488_0
- certifi=2019.3.9=py37_0
- click=7.0=py_0
- cloudpickle=0.8.0=py_0
- cytoolz=0.9.0.1=py37h470a237_1
- dask=1.1.4=py_0
- dask-core=1.1.4=py_0
- decorator=4.4.0=py_0
- distributed=1.26.0=py37_1
- freetype=2.10.0=h24853df_0
- heapdict=1.0.0=py37_1000
- ipython=7.4.0=py37h24bf2e0_0
- ipython_genutils=0.2.0=py_1
- jedi=0.13.3=py37_0
- jinja2=2.10=py_1
- jpeg=9c=h1de35cc_1001
- libblas=3.8.0=4_openblas
- libcblas=3.8.0=4_openblas
- libcxx=4.0.1=h579ed51_0
- libcxxabi=4.0.1=hebd6815_0
- libffi=3.2.1=h6de7cb9_1006
- libgfortran=3.0.1=0
- liblapack=3.8.0=4_openblas
- liblapacke=3.8.0=4_openblas
- libpng=1.6.36=ha441bb4_1000
- libtiff=4.0.10=h79f4b77_1001
- locket=0.2.0=py_2
- markupsafe=1.1.1=py37h1de35cc_0
- msgpack-python=0.6.1=py37h04f5b5a_0
- ncurses=6.1=h0a44026_1002
- numpy=1.16.2=py37hbb3c62a_1
- olefile=0.46=py_0
- openblas=0.3.5=h436c29b_1001
- openssl=1.1.1b=h01d97ff_2
- packaging=19.0=py_0
- pandas=0.24.2=py37h0a44026_0
- parso=0.3.4=py_0
- partd=0.3.9=py_0
- pexpect=4.6.0=py37_1000
- pickleshare=0.7.5=py37_1000
- pillow=5.4.1=py37hbddbef0_1000
- pip=19.0.3=py37_0
- prompt_toolkit=2.0.9=py_0
- psutil=5.6.1=py37h1de35cc_0
- ptyprocess=0.6.0=py37_1000
- pygments=2.3.1=py_0
- pyparsing=2.3.1=py_0
- python=3.7.2=hbdd33cc_0
- python-dateutil=2.8.0=py_0
- pytz=2018.9=py_0
- pyyaml=5.1=py37h1de35cc_0
- readline=7.0=hcfe32e1_1001
- scipy=1.2.1=py37hbd7caa9_1
- setuptools=40.8.0=py37_0
- six=1.12.0=py37_1000
- sortedcontainers=2.1.0=py_0
- sqlite=3.26.0=h1765d9f_1001
- tblib=1.3.2=py_1
- tk=8.6.9=ha441bb4_1000
- toolz=0.9.0=py_1
- tornado=6.0.1=py37h1de35cc_0
- traitlets=4.3.2=py37_1000
- wcwidth=0.1.7=py_1
- wheel=0.33.1=py37_0
- xz=5.2.4=h1de35cc_1001
- yaml=0.1.7=h1de35cc_1001
- zict=0.1.4=py_0
- zlib=1.2.11=h1de35cc_1004
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.