Support Serving Latest Offline Segments Immediately
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
## Overview
For hybrid setups, Pinot splits/filters broker queries to both the offline and realtime tables. It does so based on a `time value` which represents the latest available offline segments. Pinot assumes the consumer is uploading partial or intermediate segments throughout the current day, so it only serves `time value - 1` from the offline table and the rest from realtime.
As an explicit example, if I upload a segment with the date 1/2/2000, the `time value` for the table will be 1/1/2000. This means, a query for all time, will be modified to query (*, 1/1/2000] to the offline table and (1/1/2000, *) to the realtime table. In this scenario, the 1/2/2000 segment is not being served queries. Once another segment is uploaded for 1/3/2000, then the data from 1/2/2000 will be served.
## Problem Statement
In hybrid table setups where offline segments are only uploaded once per day, it is desired that those offline segments immediately start serving over the realtime segments. Today, this has to be achieved via a "hack" where an empty segment is pushed for a future date to trick Pinot into serving the latest offline segment with actual data.
## Requirements
1. This should be configurable per table.
## Design Doc
https://docs.google.com/document/d/1KykbUIstFI7F4IOOlUVnXWzpgKazwRAJdok0RT1ae64
Contributor guide
Research direction
Start with the linked design doc and the existing hybrid-table query routing and table configuration. Trace how the latest offline segment time value is selected; done means a per-table setting allows the newest uploaded offline segment to serve immediately, including the 1/2/2000 example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100