Modify the group by algorithm to work using proactive trimming logic
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
This algorithm can be extended to support group-by with ordering on keys only (no order-by on aggregated values). When there is no order-by, we can treat it as ordering on all keys. Ideally, we want to do the following improvement:
>
> * Within segment, maintain only the top `LIMIT` keys. Given the key doesn't change, we can skip adding keys when the map size is the same as `LIMIT`, and the new key is smaller than the smallest top key
> * Across segments, we can do what you are doing here
> * We don't need trim for this approach, and the result is always accurate
Thanks for the suggestion — I completely agree that the algorithm can be generalized in that direction. Will create a separate task and pick that up too
_Originally posted by @anandheritage in https://github.com/apache/pinot/issues/15844#issuecomment-2903527609_
Contributor guide
Research direction
The issue names no files, tests, or entry point. Start by tracing the existing group-by algorithm and the implementation referenced by issue #15844; confirm how ordering, LIMIT, and segment merging currently work, then verify that key-only ordering retains accurate top-LIMIT keys within and across segments without trimming.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100