apache / apache/pinot

Improve accuracy of group by without order by queries

Open
#8,039 4 comments 1 reaction 1 assignee Claimed by @vvivekiyer View on GitHub
enhancement in-progress
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently GROUP BY queries without ORDER BY can generate very inaccurate results since each server will keep at max N (N coming from LIMIT N) groups which are randomly selected and there is no resize/trimming unlike ORDER BY.

An easier way to handle this would be to add implicit ORDER BY on GROUP BY and/or agg columns if there is no ORDER BY in the query. This will allow us to reuse current ORDER BY code path which is more accurate. This will provide same levels of accuracy and determinism as current GROUP BY with ORDER BY

If we want to improve accuracy without ordering results, then some changes in TableResizer might be needed. We will continue to accumulate more records (upto `trimThreshold` like in ORDER BY code) but resizer won't sort when trimming to `trimSize`. It can simply evict `trimThreshold - trimSize` records without worrying about order. While this will improve the accuracy, the result won't be deterministic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.