doubleSum aggregator produces a result with new precision on each run
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
```json
{
"version": "0.16.1-incubating",
"modules": [
{
"name": "org.apache.druid.common.aws.AWSModule",
"artifact": "druid-aws-common",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.common.gcp.GcpModule",
"artifact": "druid-gcp-common",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.histogram.ApproximateHistogramDruidModule",
"artifact": "druid-histogram",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.theta.SketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule",
"artifact": "druid-datasketches",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.server.lookup.namespace.NamespaceExtractionModule",
"artifact": "druid-lookups-cached-global",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule",
"artifact": "mysql-metadata-storage",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.storage.s3.S3StorageDruidModule",
"artifact": "druid-s3-extensions",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.firehose.s3.S3FirehoseDruidModule",
"artifact": "druid-s3-extensions",
"version": "0.16.1-incubating"
},
{
"name": "org.apache.druid.indexing.kafka.KafkaIndexTaskModule",
"artifact": "druid-kafka-indexing-service",
"version": "0.16.1-incubating"
},
{
"name": ".bigdata.druid.HeatmapModule"
}
],
"memory": {
"maxMemory": 25746210816,
"totalMemory": 25746210816,
"freeMemory": 14935139256,
"usedMemory": 10811071560,
"directMemory": 77309411328
}
}
```
### Description
#### Query that returns a value with new precision each time
```json
{
"dataSource": "",
"queryType": "groupBy",
"intervals": [
"2019-12-12T00:00:00.000-05:00/2020-02-29T23:59:59.999-05:00"
],
"granularity": "all",
"filter": {
"type": "and",
"fields": [
{
"type": "or",
"fields": [
{
"type": "selector",
"dimension": "mid",
"value": ""
}
]
},
{
"type": "selector",
"dimension": "is_test_survey",
"value": "false"
}
]
},
"aggregations": [
{
"type": "doubleSum",
"name": "total_satisfaction_score_sum",
"fieldName": "satisfaction_score_sum"
},
{
"type": "count",
"name": "count_satisfaction_score_sum"
},
{
"type": "count",
"name": "total_respondent_count"
}
],
"postAggregations": [
{
"type": "arithmetic",
"name": "satisfaction_score",
"fields": [
{
"type": "fieldAccess",
"fieldName": "total_satisfaction_score_sum"
},
{
"type": "fieldAccess",
"fieldName": "count_satisfaction_score_sum"
}
],
"fn": "/"
}
],
"dimensions": []
}
```
Please include as much detailed information about the problem as possible.
- Steps to reproduce the problem
- Re-run the same query multiple times and it will give results with three different precisions (the value is always one of the three below)
Results:
- Run 1: Division: `50.5901529689484`, Double Sum: `3462288.888888891`
```
[
{
"version": "v1",
"timestamp": "2019-12-12T05:00:00.000Z",
"event": {
"total_respondent_count": 68438,
"count_satisfaction_score_sum": 68438,
"satisfaction_score": 50.5901529689484,
"total_satisfaction_score_sum": 3462288.888888891
}
}
]
```
- Run 2: Division: `50.590152968948395`, Double Sum: `3462288.8888888904`
```
[
{
"version": "v1",
"timestamp": "2019-12-12T05:00:00.000Z",
"event": {
"total_respondent_count": 68438,
"count_satisfaction_score_sum": 68438,
"satisfaction_score": 50.590152968948395,
"total_satisfaction_score_sum": 3462288.8888888904
}
}
]
```
- Run 3: Division: `50.59015296894839`, Double Sum: `3462288.88888889`
```
[
{
"version": "v1",
"timestamp": "2019-12-12T05:00:00.000Z",
"event": {
"total_respondent_count": 68438,
"count_satisfaction_score_sum": 68438,
"satisfaction_score": 50.59015296894839,
"total_satisfaction_score_sum": 3462288.88888889
}
}
]
```
Contributor guide
Research direction
Start by running the supplied groupBy query with the doubleSum and arithmetic post-aggregation repeatedly against the affected Druid version, comparing the three reported precisions. The issue provides no source files or tests; done means the same query no longer produces varying doubleSum and derived division precision across runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100