apache / apache/pinot

[multistage] distinct or group-by aggregate doesn't pushdown limit

Open
#11,689 2 comments 1 reaction 0 assignees View on GitHub
feature multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently:

- query similar to
```
SELECT distinct a, b FROM tbl LIMIT 10
```
will run the entire distinct set of values `(a,b)` on leaf; reshuffle based on hash-key and dedup in the intermediate stage, then finally keep 10 records at the very last stage.

- similar but a much more subtle optimization is on group-by / order-by group key with limit.
```
SELECT a, SUM(b) FROM tbl GROUP BY a ORDER BY a DESC LIMIT 10
```

a good proposal for both of these situations is to: pushdown the sorted limit all the way to the leaf stage and only keeping the limited rows before sending data out of the leaf; also potentially skip segment tasks if limits are already reach (for the order by situation it is not possible though)

Contributor guide

Open the contributing guide

Research direction

Start from the two SQL examples and trace how distinct and group-by aggregation, reshuffling, ordering, and LIMIT are handled across the leaf, intermediate, and final stages. Done means the limited rows are safely pushed toward the leaf stage for the applicable cases without changing query results, with segment-task skipping considered where the ordering allows it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.