apache / apache/pinot

Automatic, configured, and transparent index selection in Pinot queries

Open
#8,932 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

This issue is getting us into the realm of other database functionality. Because we setup so many Pinot tables and queries, we try to standardize how we select indices on all of our tables. The current logic is all dimension columns get an inverted index, the time column gets a range index, and some "key column" also gets a bloom filter.

While we expected this to be a clear trade off of cost (cost of generating and storing indices) to better latency, there are several cases where we see much 10x worse performance due to this approach. Specifically, when we have a table with ~10s millions documents, filtering by the "key column" will get us down to thousands of documents. It turns out using the inverted index on the remaining fields + filters actually causes queries to be return much more slowly than just looking up and aggregating those documents directly.

Some ideas here:
- have pinot automatically recognize these cases and stop using the index
- give users some control over what indices are used (we've had similar examples where pinot uses a startree index even though we know there's not that many records, or a range index even when we know it won't be useful for that query)
- provide information on what indices were used in the query response. without this, you have to constantly infer this information from other query stats like rows scanned in vs post filter)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Pinot's current index-selection behavior for inverted, range, bloom-filter, and star-tree indexes, along with how query statistics are exposed. Compare the proposed automatic selection, user controls, and index-use reporting ideas, then define a focused scope and measurable query-performance outcome before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.