SegmentNameGenerator: Extend interface to accept input file name
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Following Slack discussion: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1624443889243500
**Use case:**
Some tables in Pinot are used in conjunction with [IdSet filtering](https://docs.google.com/document/d/1s6DZ9eTPqH7vaKQlPjKiWb_OBC3hkkEGICIzcd5gozc/edit#) or [Lookups](https://docs.pinot.apache.org/users/user-guide-query/lookup-udf-join) - and in some cases don't have a time column.
- Hence, existing segment name generation strategies (i.e. time-based and fixed) do not allow for simple segment replacements whenever data in those "dimension" tables change (say, data for `ID` changed)
**Proposition**
Allow segment name generation to be based on the input file names such that segments can be named following a user provided id (in the file names).
E.g.
```
basedir/id1/file.parquet
basedir/id2/file.parquet
```
Would generate segments
```
_id1.segment
_id2.segment
```
Currently, the [SegmentNameGenerator interface](https://github.com/apache/incubator-pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/SegmentNameGenerator.java) doesn't allow input file names, therefore it is not possible to implement a strategy similar to the one presented above.
**Note**: If you known any other alternative to reach the use case goal, please feel free to provide ideas !
Contributor guide
Research direction
Start with pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/SegmentNameGenerator.java and review how segment naming strategies receive input. Trace the segment creation path to determine where input file names could be supplied. Done means supporting user-provided identifiers from paths such as basedir/id1/file.parquet while preserving existing naming strategies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100