Consider leveraging inverted index to augment time range index performance
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
For cases where `range` index is used for timestamps, I have observed that using a complimentary inverted index at broader time granularity than I query greatly decreases response latency. Ex. if I have epochMs values in a field, and I convert those to yyyy-MM-dd (2023-08-30) strings which are stored in a separate column with inverted index enabled, this massively improves query performance since the inverted index can efficiently produce the superset of docs that could match the epochMs range. In my case, I pad the range with an additional day on each side to allow for ignoring timezone conversions.
This might be nice to provide out of the box as a "special" range index where all of this is handled under-the-hood, at user configurable granularity (hour instead of day, for example). Might not be worth the trouble either, as it's definitely doable as-is be self-managing.
Contributor guide
Research direction
Start by reviewing Pinot's range-index and inverted-index implementations and how timestamp fields are queried. Compare the proposed configurable time granularity and timezone padding with existing index configuration and performance behavior. Done requires a maintainer-approved design and implementation scope; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100