apache / apache/druid

groupBy query: limit push down to segment scan is poor performance

Open
#9,689 8 comments 0 reactions 0 assignees View on GitHub
Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.17.1

### Description
When druid-0.17.1 is used, the performance of group by query is worse than druid-0.16.1. When
applylimitpushdowntosegment is set to false in query context, the performance returns to normal.

From this figure, we can see that the CPU consumption is in LimitedBufferHashGrouper.
![image](https://user-images.githubusercontent.com/5836707/79065418-bdfaba00-7ce2-11ea-8c86-6b4ce88b0f94.png)

query performance is according to query/time from historical log:
`{"query/time":13372,"query/bytes":6118553,"success":true,"identity":"allowAll"}`

@himanshug

Contributor guide

Open the contributing guide

Research direction

Start by comparing groupBy execution with applylimitpushdowntosegment enabled and disabled in Druid 0.17.1, using the reported query/time and CPU evidence as a baseline. Investigate LimitedBufferHashGrouper and the segment-scan limit pushdown path; done means identifying and correcting the regression, with performance returning near the 0.16.1 or disabled-setting baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.