apache / apache/druid

doubleSum aggregator produces a result with new precision on each run

Open
#9,961 0 comments 0 reactions 0 assignees View on GitHub
Area - Querying Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version
```json
{
"version": "0.16.1-incubating",
"modules": [
{
"name": "org.apache.druid.common.aws.AWSModule",
"artifact": "druid-aws-common",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.common.gcp.GcpModule",
"artifact": "druid-gcp-common",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.histogram.ApproximateHistogramDruidModule",
"artifact": "druid-histogram",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.theta.SketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.server.lookup.namespace.NamespaceExtractionModule",
"artifact": "druid-lookups-cached-global",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule",
"artifact": "mysql-metadata-storage",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.storage.s3.S3StorageDruidModule",
"artifact": "druid-s3-extensions",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.firehose.s3.S3FirehoseDruidModule",
"artifact": "druid-s3-extensions",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.indexing.kafka.KafkaIndexTaskModule",
"artifact": "druid-kafka-indexing-service",
"version": "0.16.1-incubating"
},
{
"name": ".bigdata.druid.HeatmapModule"
}
],
"memory": {
"maxMemory": 25746210816,
"totalMemory": 25746210816,
"freeMemory": 14935139256,
"usedMemory": 10811071560,
"directMemory": 77309411328
}
}
```

### Description

#### Query that returns a value with new precision each time
```json
{
"dataSource": "",
"queryType": "groupBy",
"intervals": [
"2019-12-12T00:00:00.000-05:00/2020-02-29T23:59:59.999-05:00"
],
"granularity": "all",
"filter": {
"type": "and",
"fields": [
{
"type": "or",
"fields": [
{
"type": "selector",
"dimension": "mid",
"value": ""
}
]
},
{
"type": "selector",
"dimension": "is_test_survey",
"value": "false"
}
]
},
"aggregations": [
{
"type": "doubleSum",
"name": "total_satisfaction_score_sum",
"fieldName": "satisfaction_score_sum"
},
{
"type": "count",
"name": "count_satisfaction_score_sum"
},
{
"type": "count",
"name": "total_respondent_count"
}
],
"postAggregations": [
{
"type": "arithmetic",
"name": "satisfaction_score",
"fields": [
{
"type": "fieldAccess",
"fieldName": "total_satisfaction_score_sum"
},
{
"type": "fieldAccess",
"fieldName": "count_satisfaction_score_sum"
}
],
"fn": "/"
}
],
"dimensions": []
}
```

Please include as much detailed information about the problem as possible.
- Steps to reproduce the problem
- Re-run the same query multiple times and it will give results with three different precisions (the value is always one of the three below)

Results:
- Run 1: Division: `50.5901529689484`, Double Sum: `3462288.888888891`
```
[
{
"version": "v1",
"timestamp": "2019-12-12T05:00:00.000Z",
"event": {
"total_respondent_count": 68438,
"count_satisfaction_score_sum": 68438,
"satisfaction_score": 50.5901529689484,
"total_satisfaction_score_sum": 3462288.888888891
}
}
]
```
- Run 2: Division: `50.590152968948395`, Double Sum: `3462288.8888888904`
```
[
{
"version": "v1",
"timestamp": "2019-12-12T05:00:00.000Z",
"event": {
"total_respondent_count": 68438,
"count_satisfaction_score_sum": 68438,
"satisfaction_score": 50.590152968948395,
"total_satisfaction_score_sum": 3462288.8888888904
}
}
]
```
- Run 3: Division: `50.59015296894839`, Double Sum: `3462288.88888889`
```
[
{
"version": "v1",
"timestamp": "2019-12-12T05:00:00.000Z",
"event": {
"total_respondent_count": 68438,
"count_satisfaction_score_sum": 68438,
"satisfaction_score": 50.59015296894839,
"total_satisfaction_score_sum": 3462288.88888889
}
}
]
```

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied groupBy query with the doubleSum and arithmetic post-aggregation repeatedly against the affected Druid version, comparing the three reported precisions. The issue provides no source files or tests; done means the same query no longer produces varying doubleSum and derived division precision across runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.