apache / apache/druid

Weird query results when using the DataSketches Quantiles Sketch

Open
#8,659 4 comments 2 reactions 0 assignees View on GitHub
Area - Extension
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

We used the DataSketches to compute quantiles and got very weird query results.

### Affected Version
0.12.2
### Description

Metrics spec at ingestion time:
```json
"metricsSpec": [
{
"type": "count",
"name": "count"
},
{
"type": "doubleSum",
"name": "cm_value",
"fieldName": "cm_value",
"expression": null
},
{
"type": "quantilesDoublesSketch",
"name": "cm_value_sketch",
"fieldName": "cm_value",
"k": 128
}
]
```

My query:
```json
"aggregations": [
{
"type": "quantilesDoublesSketch",
"name": "custom_value_sketch",
"fieldName": "cm_value"
},
{
"type": "doubleSum",
"name": "count",
"fieldName": "count"
},
{
"type": "doubleSum",
"name": "cm_value_sum",
"fieldName": "cm_value"
}
],
"postAggregations": [
{
"type": "quantilesDoublesSketchToQuantiles",
"name": "quantiles",
"fractions": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1
],
"field": {
"type": "fieldAccess",
"fieldName": "custom_value_sketch"
}
}
]
```

The query result:
```json
"result" : {
"count" : 4223.0,
"cm_value_sum" : 667109.0,
"quantiles" : [ 52.0, 179.0, 515.0, 929.0, 1185.0, 1426.0, 1680.0, 2047.0, 2601.0, 6000.0 ],
"custom_value_sketch" : 529
}
```
As we can see, the value of 0.5-quantile is 1185.0, so there must be nearly half of the `cm_value` greater than or equal to 1185.0. However, if we multiply 1185 and 2111 (half of the count) , we found the result is 2501535 which is much greater than the sum of cm_value 667109. Impossible! this should not be happen. We have loaded the same data into hive, and queried hive we got the result:
```json
"result" : {
"count" : 4223.0,
"cm_value_sum" : 667109.0,
"quantiles" : [ 70.0, 82.0, 96.0, 112.0, 136.0, 160.0, 189.0, 229.0, 274.8000000000002, 3368.0 ]
}
```
@AlexanderSaydakov is there any bug of DataSketches Quantiles Sketch or I used it in a wrong way?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported ingestion and query specifications for the DataSketches Quantiles Sketch in Druid, then compare the returned quantiles with the reported sum and count. Determine whether the discrepancy comes from incorrect sketch usage or a Druid bug; done means the cause is confirmed and the issue is fixed or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.