apache / apache/druid

BUG when sorting (realtime) topN results by postAggregation vs aggregation

Open
#6,375 2 comments 4 reactions 0 assignees View on GitHub
Bug stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

I'm running a simple query to get the trend of pageviews on recent data (middle manager):
```
{
"queryType":"topN",
"dataSource":"pageviews",
"dimension":"onlineId",
"metric": {
"type": "numeric",
"metric": "score"
},
"granularity": {"type": "duration", "duration": 1900000, "origin": "2018-09-25T09:12:00.000Z"},
"threshold": 60,
"intervals":[
"2018-09-25T09:12:00.000Z/PT30M"
],
"aggregations":[
{
"type":"filtered",
"filter":{
"type":"interval",
"dimension":"__time",
"intervals":[
"2018-09-25T09:12:00.000Z/PT15M"
]
},
"aggregator":{
"type":"longSum",
"name":"total_old",
"fieldName":"count"
}
},
{
"type":"filtered",
"filter":{
"type":"interval",
"dimension":"__time",
"intervals":[
"2018-09-25T09:27:00.000Z/PT15M"
]
},
"aggregator":{
"type":"longSum",
"name":"total_new",
"fieldName":"count"
}
}
],
"postAggregations":[
{
"type":"arithmetic",
"name":"score",
"fn":"-",
"fields":[
{
"type":"fieldAccess",
"fieldName":"total_new"
},
{
"type":"fieldAccess",
"fieldName":"total_old"
}
]
}
]
}
```

The outcomes vary based on which metric I select for sorting. "score" returns wrong values, while "total_new" returns correct results.

Example Results:

Sorted by postAggregation ``` "metric":"score" ```
```
"result": [
{
"total_old": 344,
"onlineId": "10264391",
"total_new": 1424,
"score": 1080
},
{
"total_old": 134,
"onlineId": "6372612",
"total_new": 606,
"score": 472
},
{
"total_old": 12,
"onlineId": "10271038",
"total_new": 263,
"score": 251
},
{
"total_old": 53,
"onlineId": "10261042",
"total_new": 285,
"score": 232
},
...
]
```

Sorted by aggregation ``` "metric":"total_new" ```
```
"result": [
{
"total_old": 1250,
"onlineId": "10264391",
"total_new": 1424,
"score": 174
},
{
"total_old": 421,
"onlineId": "6372612",
"total_new": 606,
"score": 185
},
{
"total_old": 408,
"onlineId": "10271038",
"total_new": 360,
"score": -48
},
{
"total_old": 251,
"onlineId": "10261042",
"total_new": 285,
"score": 34
},
...
]
```

Notes:
- Druid v0.12.3
- Data is ingested from Kafka
- When I set the threshold high enough, both sortings return the correct results
- When I query data from the historicals (-1day), both sortings return the correct results

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the topN query against the realtime MiddleManager data described in the report, comparing sorting by the postAggregation score with sorting by total_new. Check how thresholding affects the results and compare them with the historicals query; done means both sort metrics return consistent values while preserving the reported historical behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.