Discrepancy with null in APPROX_COUNT_DISTINCT_BUILTIN between MSQ and native
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
While resolving some test failures, I noticed that there is a small discrepancy in how APPROX_COUNT_DISTINCT_BUILTIN works. This only occurs if `druid.generic.useDefaultValueForNull` is false and there are null values present in the segment queried.
For a datasource `foo` which contains the following values in dim2
```
"a"
null
""
"a"
"abc"
null
```
On running a query `SELECT dim2, APPROX_COUNT_DISTINCT_BUILTIN(dim2) FROM druid.foo GROUP BY 1`, we get the following results:
Native:
```
null, 0L
"", 1L
"a", 1L
"abc", 1L
```
MSQ:
```
null, 0L
"", 0L
"a", 1L
"abc", 1L
```
MSQ seems to ignore the empty string in the same way as null, while native seems to have the correct behaviour. A change might need to be made to bring MSQ in line.
Contributor guide
Research direction
Start by reproducing the APPROX_COUNT_DISTINCT_BUILTIN query with druid.generic.useDefaultValueForNull set to false and the sample dim2 values, then compare native and MSQ results. The work is done when MSQ treats the empty string as a value, returning 1L for it like native, while still returning 0L for null.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100