apache / apache/pinot

Consider leveraging inverted index to augment time range index performance

Open
#11,464 4 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

For cases where `range` index is used for timestamps, I have observed that using a complimentary inverted index at broader time granularity than I query greatly decreases response latency. Ex. if I have epochMs values in a field, and I convert those to yyyy-MM-dd (2023-08-30) strings which are stored in a separate column with inverted index enabled, this massively improves query performance since the inverted index can efficiently produce the superset of docs that could match the epochMs range. In my case, I pad the range with an additional day on each side to allow for ignoring timezone conversions.

This might be nice to provide out of the box as a "special" range index where all of this is handled under-the-hood, at user configurable granularity (hour instead of day, for example). Might not be worth the trouble either, as it's definitely doable as-is be self-managing.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Pinot's range-index and inverted-index implementations and how timestamp fields are queried. Compare the proposed configurable time granularity and timezone padding with existing index configuration and performance behavior. Done requires a maintainer-approved design and implementation scope; the issue names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.