Optimize multiple COUNT(DISTINCT) memory via logical plan rewrite
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion's default execution creates separate HashSet accumulators for each distinct count per group. With high-cardinality data (eg sellers with 4,000+ distinct cities in an ecommerce dataset), this causes memory explosion.
### Describe the solution you'd like
DataFusion already optimizes the single shared distinct field case via SingleDistinctToGroupBy. This PR adds a conservative logical rewrite for multiple distinct COUNT(DISTINCT …) arguments by splitting work into per-distinct branches joined on the group keys, which reduces peak memory for eligible plans.
COUNT(DISTINCT x) must ignore NULL x; the rewrite applies x IS NOT NULL on each distinct branch before inner grouping so semantics stay aligned with count_distinct behavior.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by locating the existing SingleDistinctToGroupBy logical rewrite and the optimizer paths for grouped COUNT(DISTINCT ...) expressions. Trace how eligible multiple-distinct plans could split into per-distinct branches joined on group keys, preserving the required IS NOT NULL filtering. Done means eligible plans reduce peak distinct-accumulator memory while COUNT(DISTINCT x) retains its NULL-ignoring semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100