`are_co_aligned` is tokenizing too greedily causing to be possibly slow
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
The utility function `are_co_aligned`, see
https://github.com/dask-contrib/dask-expr/blob/9334e062a7b41161977ca1c42176197629569cc5/dask_expr/_expr.py#L2863-L2874
is unfortunately rather slow due to the tokenization and lack of caching. in paruqet_reader benchmarks on larger datasets, I saw this slowing down the optimize step by almost a second (when using pyarrowFS such that filters are pushed down

On top of this, I believe the implementation is unsafe since it is putting `Expr` objects into a `set`. Sets and dicts are requiring both `__hash__` and `__eq__` to be implemented and working as the stdlib protocol defines them. While this is true for hash (it hashes the name, this is not the case for `__eq__` since this just creates another `Expr` instance instead of returning a bool. I suspect this just tells how the set is redundant if there hasn't been ever a hash collision / duplicate object here.
Contributor guide
Research direction
Start with dask_expr/_expr.py around are_co_aligned at lines 2863-2874, then reproduce the parquet_reader optimization benchmark described in the issue on a larger dataset. Check how tokenization and the Expr set affect the optimize step and verify that the completed change avoids the slowdown and unsafe set behavior without changing alignment results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100