apache / apache/pinot

Modify the group by algorithm to work using proactive trimming logic

Open
#15,896 2 comments 0 reactions 0 assignees View on GitHub
enhancement query stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

This algorithm can be extended to support group-by with ordering on keys only (no order-by on aggregated values). When there is no order-by, we can treat it as ordering on all keys. Ideally, we want to do the following improvement:
>
> * Within segment, maintain only the top `LIMIT` keys. Given the key doesn't change, we can skip adding keys when the map size is the same as `LIMIT`, and the new key is smaller than the smallest top key
> * Across segments, we can do what you are doing here
> * We don't need trim for this approach, and the result is always accurate

Thanks for the suggestion — I completely agree that the algorithm can be generalized in that direction. Will create a separate task and pick that up too

_Originally posted by @anandheritage in https://github.com/apache/pinot/issues/15844#issuecomment-2903527609_

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry point. Start by tracing the existing group-by algorithm and the implementation referenced by issue #15844; confirm how ordering, LIMIT, and segment merging currently work, then verify that key-only ordering retains accurate top-LIMIT keys within and across segments without trimming.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.