apache / apache/datafusion-comet

Add support for `COUNT(DISTINCT expr, expr1, ...)`

Open
#2,292 2 comments 2 reactions 1 assignee Assigned to @comphead View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

The expression `COUNT(DISTINCT expr)` is relatively common and it is used in TPC-H, so it would be good to be able to accelerate this in Comet.

Spark supports multiple expressions e.g. `COUNT(DISTINCT a, b, c)`, but DataFusion does not, so we should only attempt to accelerate this if there is a single input expression.

Implementing this feature is not trivial because there are some design issues with how we currently support partial aggregates. Specifically, we do not report the correct output schema from the partial aggregate. For the aggregate expressions that we currently support it doesn't matter because the output of the partial and final aggregates is the same. For example `SUM(int_column)` will have the output type `int` for both partial and final. For `COUNT(DISTINCT int_column)` the output of the partial will be a **list** of int and the output of the final will be a long.

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.