dask / dask/dask-expr

`are_co_aligned` is tokenizing too greedily causing to be possibly slow

Open
#907 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

The utility function `are_co_aligned`, see

https://github.com/dask-contrib/dask-expr/blob/9334e062a7b41161977ca1c42176197629569cc5/dask_expr/_expr.py#L2863-L2874

is unfortunately rather slow due to the tokenization and lack of caching. in paruqet_reader benchmarks on larger datasets, I saw this slowing down the optimize step by almost a second (when using pyarrowFS such that filters are pushed down

![image](https://github.com/dask-contrib/dask-expr/assets/8629629/15cf2cef-48e3-4ebe-9468-eeaa51969e0a)

On top of this, I believe the implementation is unsafe since it is putting `Expr` objects into a `set`. Sets and dicts are requiring both `__hash__` and `__eq__` to be implemented and working as the stdlib protocol defines them. While this is true for hash (it hashes the name, this is not the case for `__eq__` since this just creates another `Expr` instance instead of returning a bool. I suspect this just tells how the set is redundant if there hasn't been ever a hash collision / duplicate object here.

Contributor guide

Open the contributing guide

Research direction

Start with dask_expr/_expr.py around are_co_aligned at lines 2863-2874, then reproduce the parquet_reader optimization benchmark described in the issue on a larger dataset. Check how tokenization and the Expr set affect the optimize step and verify that the completed change avoids the slowdown and unsafe set behavior without changing alignment results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.