Remove dependence of strict naming convention for realtime segments
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
The realtime segments are named as "tableName__partitionId__sequenceNum__timestamp".
This convention makes it harder to perform operations such as table migration, adhoc uploads, etc.
In other intiatives such as pluggable streams, we have encountered issues with the presence of timestamp in segment name.
It would be nice to get to a state where we dont depend on the segment name having a convention as such. For starters, we can make this change for completed segments, and then trickle it down to consuming as well.
cc @mayankshriv
Contributor guide
Research direction
Start by tracing how realtime segment names are created and consumed, focusing first on completed segments and then on consuming segments. Done means these operations no longer depend on the tableName__partitionId__sequenceNum__timestamp convention, including table migration and ad hoc uploads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100