apache / apache/pinot

COUNT DISTINCT on multiple columns produces wrong result.

Open
#7,560 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

The query `SELECT count(DISTINCT name, score) FROM scores` produces wrong results. This happens because all arguments except the first one are ignored while constructing DISTINCTCOUNT Function in `AggregationFunctionFactory.java`.

One way to fix this is to using CONCAT function to concat all arguments into a single argument before creating the DISTINCTCOUNT function. Does this sound good? If so I will go ahead and put this in a PR. Any other suggestions?

```
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionFactory.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionFactory.java
index 328560739d..b9e78fe098 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionFactory.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionFactory.java
@@ -159,7 +159,13 @@ public class AggregationFunctionFactory {
case MINMAXRANGE:
return new MinMaxRangeAggregationFunction(firstArgument);
case DISTINCTCOUNT:
- return new DistinctCountAggregationFunction(firstArgument);
+ if (arguments.size() == 1) {
+ return new DistinctCountAggregationFunction(firstArgument);
+ }
+
+ arguments.add(ExpressionContext.forLiteral(""));
+ return new DistinctCountAggregationFunction(ExpressionContext
+ .forFunction(new FunctionContext(FunctionContext.Type.TRANSFORM, "CONCAT", arguments)));
case DISTINCTCOUNTBITMAP:
return new DistinctCountBitmapAggregationFunction(firstArgument);
case SEGMENTPARTITIONEDDISTINCTCOUNT:
```

Contributor guide

Open the contributing guide

Research direction

Start in pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionFactory.java and reproduce SELECT count(DISTINCT name, score) FROM scores. Trace how DISTINCTCOUNT is built when multiple arguments are supplied and verify that the result accounts for every argument. Done means the query returns the correct distinct count without ignoring later arguments.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.