Support SegmentSelector API to select segment names based on expression
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Since Pinot supports batch segments replacement, there is a need to find out what are segments to replace in an existing table based on segment metadata.
E.g. When we want to backfill data on a particular date, we want to find out all the segments for that date.
Sample expression:
1. Find segments for a given time range:
`segment.start.time >= 1614000000 and segment.end.time <1614150000`
2. Find segments for a partition:
`segment.partition.id==1`
Ideally, if we can have a system logical table like `__SEGMENTS_METADATA` then we can direct send a query like:
```
SELECT segment_name FROM __SEGMENTS_METADATA WHERE SEGMENT_START_TIME >= 1614000000 and SEGMENT_END_TIME <1614150000
```
to query segment metadata.
Contributor guide
Research direction
Start by locating the SegmentSelector API and the segment metadata query or expression entry points in Pinot. Compare the time-range and partition examples in this issue with the existing metadata model, then verify that segment names can be selected for both expressions and the proposed logical-table query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100