Realtime to Offline segment conversion: Provision for rolling offline segments based on segment size
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
REALTIME tables segment size can be controlled using `realtime.segment.flush.threshold.segment.size` (say "100M")
We don't have any similar configuration for OFFLINE tables which can be used during Realtime to Offline table conversion.
There is a configuration `maxNumRecordsPerSegment` with which we can indirectly achieve desired size. However, having a direct size based control will be really helpful and straight forward.
Contributor guide
Research direction
Start by tracing the realtime-to-offline table conversion and the existing maxNumRecordsPerSegment configuration. Compare that path with the realtime.segment.flush.threshold.segment.size setting, then verify that offline conversion supports a direct segment-size control and that the resulting segments follow the configured limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100