dask / dask/dask

Plan for high level graph optimizations

Open
#5,644 10 comments 0 reactions 0 assignees View on GitHub
highlevelgraph
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

Now that we have high level graphs in our collections, we can do some more complex optimizations. We've done a little bit of this with blockwise fusion, read_parquet and column selection, and root fusion, all of which have had really positive effects.

There is likely more that we can do here. Some thoughts ...

- Fuse column selection with other kinds of data access, like read_csv or ORC
- Pass slicing through some blockwise operations in Dask array (a long-held request by @shoyer)
- Optimize the subgraph callables in blockwise with Numba (or something else) for dask array to avoid memory copies and maybe reduce serialization time
- Swap around Joins and filters and column access
- ...

It would be nice to have a current maintainer consider these options, guess how long they would take, and see what makes sense to do short term

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.