apache / apache/druid

Discrepancy with null in APPROX_COUNT_DISTINCT_BUILTIN between MSQ and native

Open
#13,950 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

While resolving some test failures, I noticed that there is a small discrepancy in how APPROX_COUNT_DISTINCT_BUILTIN works. This only occurs if `druid.generic.useDefaultValueForNull` is false and there are null values present in the segment queried.

For a datasource `foo` which contains the following values in dim2

```
"a"
null
""
"a"
"abc"
null
```
On running a query `SELECT dim2, APPROX_COUNT_DISTINCT_BUILTIN(dim2) FROM druid.foo GROUP BY 1`, we get the following results:
Native:
```
null, 0L
"", 1L
"a", 1L
"abc", 1L
```
MSQ:
```
null, 0L
"", 0L
"a", 1L
"abc", 1L
```
MSQ seems to ignore the empty string in the same way as null, while native seems to have the correct behaviour. A change might need to be made to bring MSQ in line.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the APPROX_COUNT_DISTINCT_BUILTIN query with druid.generic.useDefaultValueForNull set to false and the sample dim2 values, then compare native and MSQ results. The work is done when MSQ treats the empty string as a value, returning 1L for it like native, while still returning 0L for null.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.