apache / apache/pinot

Remove dependence of strict naming convention for realtime segments

Open
#7,594 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

The realtime segments are named as "tableName__partitionId__sequenceNum__timestamp".
This convention makes it harder to perform operations such as table migration, adhoc uploads, etc.
In other intiatives such as pluggable streams, we have encountered issues with the presence of timestamp in segment name.

It would be nice to get to a state where we dont depend on the segment name having a convention as such. For starters, we can make this change for completed segments, and then trickle it down to consuming as well.

cc @mayankshriv

Contributor guide

Open the contributing guide

Research direction

Start by tracing how realtime segment names are created and consumed, focusing first on completed segments and then on consuming segments. Done means these operations no longer depend on the tableName__partitionId__sequenceNum__timestamp convention, including table migration and ad hoc uploads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.