apache / apache/pinot

Support SegmentSelector API to select segment names based on expression

Open
#6,606 4 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Since Pinot supports batch segments replacement, there is a need to find out what are segments to replace in an existing table based on segment metadata.

E.g. When we want to backfill data on a particular date, we want to find out all the segments for that date.

Sample expression:
1. Find segments for a given time range:
`segment.start.time >= 1614000000 and segment.end.time <1614150000`
2. Find segments for a partition:
`segment.partition.id==1`

Ideally, if we can have a system logical table like `__SEGMENTS_METADATA` then we can direct send a query like:
```
SELECT segment_name FROM __SEGMENTS_METADATA WHERE SEGMENT_START_TIME >= 1614000000 and SEGMENT_END_TIME <1614150000
```
to query segment metadata.

Contributor guide

Open the contributing guide

Research direction

Start by locating the SegmentSelector API and the segment metadata query or expression entry points in Pinot. Compare the time-range and partition examples in this issue with the existing metadata model, then verify that segment names can be selected for both expressions and the proposed logical-table query.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.