apache / apache/pinot

Star-tree Optimizations

Open
#7,137 10 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Problems:
1. Certain predicates (e.g. `NOT_EQUAL`) can select too many branches, limiting the performance of star-tree
2. Star-tree does not support filters with `OR`

Potential solutions for 1:
- During aggregation, negate the aggregation result of the non-selected branches from the star node branch. This solution could become complicated for multiple predicates.
- Create a derived column for the predicate result, and apply star-tree on top of it. This solution can only work on the fixed predicate.
- First use inverted index to do the filtering, then use pre-aggregated docs to cover the contiguous selected docs

Potential solution for 2:
- Use bitmap to store the selected docs for each predicate under `OR`, then union the bitmaps

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the star-tree predicate and aggregation handling, then compare the two listed problem areas and their potential solutions; the issue does not define an acceptance criterion for when either optimization is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.