apache / apache/pinot

Support Serving Latest Offline Segments Immediately

Open
#5,751 17 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

## Overview

For hybrid setups, Pinot splits/filters broker queries to both the offline and realtime tables. It does so based on a `time value` which represents the latest available offline segments. Pinot assumes the consumer is uploading partial or intermediate segments throughout the current day, so it only serves `time value - 1` from the offline table and the rest from realtime.

As an explicit example, if I upload a segment with the date 1/2/2000, the `time value` for the table will be 1/1/2000. This means, a query for all time, will be modified to query (*, 1/1/2000] to the offline table and (1/1/2000, *) to the realtime table. In this scenario, the 1/2/2000 segment is not being served queries. Once another segment is uploaded for 1/3/2000, then the data from 1/2/2000 will be served.

## Problem Statement

In hybrid table setups where offline segments are only uploaded once per day, it is desired that those offline segments immediately start serving over the realtime segments. Today, this has to be achieved via a "hack" where an empty segment is pushed for a future date to trick Pinot into serving the latest offline segment with actual data.

## Requirements

1. This should be configurable per table.

## Design Doc

https://docs.google.com/document/d/1KykbUIstFI7F4IOOlUVnXWzpgKazwRAJdok0RT1ae64

Contributor guide

Open the contributing guide

Research direction

Start with the linked design doc and the existing hybrid-table query routing and table configuration. Trace how the latest offline segment time value is selected; done means a per-table setting allows the newest uploaded offline segment to serve immediately, including the 1/2/2000 example.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.