Add MultiDistinctToCrossJoin optimizer rule for parallel distinct aggregates
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Queries like `count(DISTINCT hits.SearchPhrase), count(DISTINCT hits.MobilePhone), count(DISTINCT hits.MobilePhoneModel)` still don't run in parallel as they can not be optimized using the `SingleDistinctToGroupBy` optimization.
### Describe the solution you'd like
Change it to issue the distinct queries in a separate stage, joining the results (either using join or union but join has a advantage on memory usage and cache locality as it runs them one by one).
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by tracing the existing SingleDistinctToGroupBy optimizer rule and the handling of multiple DISTINCT aggregates. Compare the proposed join and union approaches, then validate that the new MultiDistinctToCrossJoin rule produces parallel distinct stages with correctly joined results for the example query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100