Performance degradation in topN queries when SQL-compatible null handling is enabled
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.16.0, 0.17.0
### Description
Given a Druid v0.16.0 or v0.17.0 cluster configured with SQL-compatible null handling enabled,
the intial performance we measured was inconspicuous, but after a while, there would
be a drastic performance degradation for topN queries.
After much testing we found out that initial performance of a freshly started Druid
cluster would be consistently fast UNTIL a groupbyV2 query gets exececuted for the
first time.
After that the performance of topN queries with numerical sort would degrade by 70% or more.
This degradation is specific to topN queries and also seems to apply
only to heavy topN queries (8 aggregations, several sequential passes).
Our tests are performed on a Druid cluster that exclusively holds segments produced with prior Druid versions or with SQL-compatible null handling disabled.
We looked at any operational metric we have but could not find a root cause for the degradation.
The degradation would not fade out with time. Also, a forced full garbage collection would not
recover any performance.
Furthermore the execution of a single groupbV2 query, any groupbyV2 query seems to trigger the
degradation.
We have a performance testsuite and a metrics dashboard.
In the screenshots from the perf testsuite below you can see the degradation
in topn queries after the execution of the first groupbyV2 query in the before/after
view.
Furthermore, the dashboard shows a different test we performed to illustrate the
performance degradation: we initially sent a sequential stream of topN queries to a
freshly started Druid cluster for a long time. Then we issued a single groupbyV2 query
while the stream of topN queries would continue. One can clearly see how performance
degrades immediately and is constant before and after.
The dashboard shows the segment-scan-time metrics to illustrate that the degradation
happens on the historicals by way of decreased scan times.
In the attempt to hone in on root-causes, we ran further tests that had subsystems of Druid disabled:
* disabling metric emission
* disabling log emission
* disabling all caches
However, in all these cases the performance degradation remained.
As we keep sending the same query many times, we can also rule out effects caused by disk access because the segments needed for serving the query would be paged into memory.
Then we turned off the SQL-compatible null handling and the performance issue was gone.


UPDATE: meanwhile we have updated our Druid cluster to version 0.17.0 and reran our tests and can confirm that the performance issue still remains.
Following is an example query which would have a degraded performance after the first execution of a groupbyV2 query:
`
{
"queryType": "topN",
"dataSource": "supply-activities",
"intervals": "2016-12-03T00/2016-12-04T00",
"context": {
"timeout": 180000,
"priority": 0,
"useCache": false,
"populateCache": false,
"useResultLevelCache": false,
"populateResultLevelCache": false,
"minTopNThreshold": 1000,
"vectorize": "false"
},
"dimension": "adSpaceId",
"metric": {
"type": "numeric",
"metric": "interactions"
},
"threshold": 50,
"granularity": "all",
"aggregations": [
{
"name": "auctionsWon",
"type": "doubleSum",
"fieldName": "auctionCount"
},
{
"name": "views",
"type": "longSum",
"fieldName": "view"
},
{
"name": "clicks",
"type": "longSum",
"fieldName": "click"
},
{
"name": "interactions",
"type": "longSum",
"fieldName": "interactionCount"
},
{
"name": "validInteractions",
"type": "longSum",
"fieldName": "validInteractionCount"
},
{
"name": "erroneousInteractions",
"type": "longSum",
"fieldName": "erroneousInteractionCount"
},
{
"name": "noAdInteractions",
"type": "longSum",
"fieldName": "noAdInteractionCount"
},
{
"name": "accountableRevenue",
"type": "doubleSum",
"fieldName": "accountableRevenue"
}
]
}
`
Contributor guide
Research direction
Start by reproducing the supplied topN query on Druid 0.16.0 or 0.17.0 with SQL-compatible null handling, then run a groupbyV2 query and compare performance using the mentioned performance testsuite and metrics dashboard. Done means identifying and fixing the degradation so topN performance remains stable after groupbyV2 execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100