apache / apache/datafusion

Optimize multiple COUNT(DISTINCT) memory via logical plan rewrite

Open
#21,087 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion's default execution creates separate HashSet accumulators for each distinct count per group. With high-cardinality data (eg sellers with 4,000+ distinct cities in an ecommerce dataset), this causes memory explosion.

### Describe the solution you'd like

DataFusion already optimizes the single shared distinct field case via SingleDistinctToGroupBy. This PR adds a conservative logical rewrite for multiple distinct COUNT(DISTINCT …) arguments by splitting work into per-distinct branches joined on the group keys, which reduces peak memory for eligible plans.

COUNT(DISTINCT x) must ignore NULL x; the rewrite applies x IS NOT NULL on each distinct branch before inner grouping so semantics stay aligned with count_distinct behavior.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing SingleDistinctToGroupBy logical rewrite and the optimizer paths for grouped COUNT(DISTINCT ...) expressions. Trace how eligible multiple-distinct plans could split into per-distinct branches joined on group keys, preserving the required IS NOT NULL filtering. Done means eligible plans reduce peak distinct-accumulator memory while COUNT(DISTINCT x) retains its NULL-ignoring semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.