apache / apache/druid

SQL: top-level sketch functions do not return sketches

Open
#9,419 6 comments 2 reactions 0 assignees View on GitHub
Area - Querying Area - SQL Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version
0.17.0

### Description
The following query should be returning materialized sketches in the form of base64 encoded strings such that the results can be exported, forklifted and later on used in other contexts.
The Druid documentation states that the following three functions would return sketches, but if they are used at top-level as in the following example, they either return counts or unique counts but no sketches:

```
SELECT
DS_HLL(countryName),
DS_THETA(countryName),
DS_QUANTILES_SKETCH(countryName)
FROM wikipedia
```
This yields
105, 105.0, 2383

The documentation states:

> DS_HLL(expr, [lgK, tgtHllType]) | Creates an HLL sketch on the values of expr

> DS_THETA(expr, [size]) | Creates a Theta sketch on the values of expr

> DS_QUANTILES_SKETCH(expr, [k]) | Creates a Quantiles sketch on the values of expr

INTERESTINGLY, the output of the above functions also has a co-dependence on whether there are further projections in the query:

The following query produces outputs that are numbers:
```
SELECT
DS_QUANTILES_SKETCH(countryName, 8),
DS_QUANTILES_SKETCH(countryName, 64),
DS_QUANTILES_SKETCH(countryName),
SUM(sum_added)
FROM wikipedia
```
|| col1 || col2 || col3 || col4 ||
| 2383 | 2383 | 2383 | 11774265 |

But when adding further projections, the output of these first three projections now changes:

```
SELECT
DS_QUANTILES_SKETCH(countryName, 8),
DS_QUANTILES_SKETCH(countryName, 64),
DS_QUANTILES_SKETCH(countryName),
DS_GET_QUANTILES(DS_QUANTILES_SKETCH(countryName), 0.1, 0.3, 0.5, 0.7, 0.9),
SUM(sum_added)
FROM wikipedia
```

now the output is:

|| col1 || col2 || col3 || col4 || col5 ||
| AgMIGggAA... | AgMIGggAA... | AgMIGggAA... | [0.0,0.0,0.0,0.0,0.0] | 11774265 |

So the output of the first expression "DS_QUANTILES_SKETCH(countryName, 8)" has changed just because another column expression got added to the query

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two SQL queries against Druid 0.17.0 and compare the top-level results of DS_HLL, DS_THETA, and DS_QUANTILES_SKETCH with and without the additional projection. Trace the sketch functions and DS_GET_QUANTILES entry points to determine why output changes; done means the sketch functions consistently return exportable base64-encoded sketches at top level.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.